Geoportal Helpdesk - Support #2509 AT - LFRZ: Observed reduction in the number of metadata records 06 Oct 2015 11:10 am - Angelo Quaglia Status: Resolved Start date: 06 Oct 2015 Priority: Normal Due date: Assignee: Angelo Quaglia % Done: 0% Category: Harvesting results Estimated time: 0.00 hour Target version: Submitting Organisation: AT - LFRZ Country: AT - Austria Proactive: Yes Originating UI: Description From: Angelo Quaglia [mailto:[email protected]] Sent: 06 October 2015 11:09 To: Seeger, Oliver; Melzer, Georg Cc: 'Infrastruktur und Betrieb' Subject: Observed reduction in the number of metadata records Importance: High Dear Sirs, We have observed a decrease in the number of metadata resources returned by the National Discovery Service of Austria. They went from 846 down to 828. Is this expected? The reason for this enquiry is that we are in the process of reconciling what is declared in the annual Monitoring & Reporting documentation with what is actually by returned by Member States Discovery Services and we would like to be sure that what we receive is actually what intended. Latest report: http://inspire-geoportal.ec.europa.eu/resources/errors/INSPIRE-1b978242-062b-11e1-b7de-52540004b857_20151006-055211/servi es/1/PullResults/ /system/rich/rich_files/rich_files/000/000/341/original/mail248.png http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-1b978242-062b-11e1-b7de-52540004b857_20151004-190027/services/1/P llResults/ 09 Jul 2018 1/25
Transcript
Geoportal Helpdesk - Support #2509AT - LFRZ: Observed reduction in the number of metadata records06 Oct 2015 11:10 am - Angelo Quaglia
Status: Resolved Start date: 06 Oct 2015Priority: Normal Due date:Assignee: Angelo Quaglia % Done: 0%Category: Harvesting results Estimated time: 0.00 hourTarget version:Submitting Organisation:AT - LFRZ Country: AT - AustriaProactive: Yes Originating UI:Description
From: Angelo Quaglia [mailto:[email protected]]Sent: 06 October 2015 11:09To: Seeger, Oliver; Melzer, GeorgCc: 'Infrastruktur und Betrieb'Subject: Observed reduction in the number of metadata recordsImportance: High
Dear Sirs,
We have observed a decrease in the number of metadata resources returned by the National Discovery Service of Austria.
They went from 846 down to 828.
Is this expected?
The reason for this enquiry is that we are in the process of reconciling what is declared in the annual Monitoring & Reportingdocumentation with what is actually by returned by Member States Discovery Services and we would like to be sure that what wereceive is actually what intended.
The views expressed are purely those of the writer and may not in any circumstances be regarded as stating an official position of theEuropean Commission.
Related issues:Copied to Geoportal Helpdesk - Support # 2513: AT - LFRZ: Access to the Disco... Closed 06 Oct 2015
History#1 - 06 Oct 2015 04:51 pm - Angelo Quaglia
From: Illmeyer, Manuel [mailto:[email protected]]Sent: 06 October 2015 15:58To:[email protected]: AW: Observed reduction in the number of metadata records
Klassifizierung - INTERN: Die Inhalte und Informationen dieser Nachricht sind ausschließlich für die kommunizierenden Unternehmen bestimmt.
we have analysed the reports from 2015-10-06 and 2015-10-04 and found out, that there are 18 datasets which are twice in the report from2015-10-04. There only exist 828 distinct IDs in both reports.
The duplicate metadata have following IDs (details in the csv attached):
Diese Nachricht und eventuelle Dateianhänge sind vertraulich und nur für den (die) darin angegebenen Adressaten bestimmt. Jeder Gebrauch, jedeWeitergabe oder Vervielfältigung durch andere als den (die) bestimmten Adressaten ist verboten. Sollten Sie diese Sendung irrtümlich erhalten haben,verständigen Sie bitte unverzüglich per E-Mail den Absender und löschen Sie die Sendung samt allfälligen Dateianhängen aus Ihrem System.
Please note:
This message and any attachment thereof are confidential and may be privileged or otherwise protected from disclosure. If you are not the intendedrecipient any use, distribution, copying or disclosure is strictly prohibited. If you have received this message in error, please notify the senderimmediately by return e-mail and delete this message and any attachment thereof from your system.
#2 - 06 Oct 2015 04:52 pm - Angelo Quaglia
From: Angelo Quaglia [mailto:[email protected]]Sent: 06 October 2015 16:51To: 'Illmeyer, Manuel'Subject: RE: Observed reduction in the number of metadata records
Dear Manuel,
Many thanks for your fast and detailed response.
During the harvesting documented in the report from 2015-10-04 the fileIdentifier 0cab4580-e202-4e2f-9edc-5699db618535 was present in twodifferent metadata records:
I assume the latest harvesting is not affected by the same issues so please confirm I should promote it to production.
Best regards,
Angelo
#3 - 06 Oct 2015 05:11 pm - Angelo Quaglia
From: Angelo Quaglia [mailto:[email protected]]Sent: 06 October 2015 17:11To: 'Illmeyer, Manuel'Subject: RE: Observed reduction in the number of metadata records
Dear Manuel,
There were multiple datasets and multiple services metadata sharing the same fileIdentifier:
1036cbad-4f03-4ddd-b825-06ccc844573 – two datasets metadata found
From: Illmeyer, Manuel [mailto:[email protected]]Sent: 07 October 2015 07:49To: Angelo QuagliaSubject: AW: Observed reduction in the number of metadata records
Klassifizierung - INTERN: Die Inhalte und Informationen dieser Nachricht sind ausschließlich für die kommunizierenden Unternehmen bestimmt.
Dear Angelo,
thanks for your response. The thing is, we only have these MD_Metadata elements once in our national discovery service. The pull results withidentical file identifiers are the exact same metadata records. So I don’t know if this was a problem with our discovery service or a problem with theGetRecordsResponse.
But since the problem didn’t occur on the Pull from 2015-10-06, you can promote it to production.
Best Regards,
Manuel
#5 - 07 Oct 2015 09:12 am - Angelo Quaglia- Status changed from Assigned to Resolved
From: Angelo Quaglia [mailto:[email protected]]Sent: 07 October 2015 09:12To: 'Illmeyer, Manuel'Subject: RE: Observed reduction in the number of metadata records
Dear Manuel,
It’s done.
Best regards,
Angelo
09 Jul 2018 8/25
#6 - 13 Oct 2015 10:05 am - Angelo Quaglia- Status changed from Resolved to Feedback
From: Angelo Quaglia [mailto:[email protected]]Sent: 13 October 2015 10:04To: 'Illmeyer, Manuel'Subject: RE: Observed reduction in the number of metadata records
Dear Manuel,
The problem is occurring again.
The number of metadata records returned keeps changing, depending on when the harvesting is done:
The views expressed are purely those of the writer and may not in any circumstances be regarded as stating an official position of the EuropeanCommission.
#7 - 13 Oct 2015 06:32 pm - Angelo Quaglia
From: Illmeyer, Manuel [mailto:[email protected]]Sent: 13 October 2015 14:45To: Angelo QuagliaSubject: AW: Observed reduction in the number of metadata records
Klassifizierung - INTERN: Die Inhalte und Informationen dieser Nachricht sind ausschließlich für die kommunizierenden Unternehmen bestimmt.
Dear Angelo,
thanks for the information. There are some changed metadata elements but there are also duplicate metadata uuids in our discovery service. We willinvestigate now, why these elements are delivered twice.
Best regards,
Manuel
#8 - 14 Oct 2015 08:57 am - Angelo Quaglia
From: Illmeyer, Manuel [mailto:[email protected]]Sent: 14 October 2015 07:48To: Angelo QuagliaSubject: AW: Observed reduction in the number of metadata records
Klassifizierung - INTERN: Die Inhalte und Informationen dieser Nachricht sind ausschließlich für die kommunizierenden Unternehmen bestimmt.
Dear Angelo,
we have made some improvements on the discovery service. I hope this problem won’t occur again.
09 Jul 2018 11/25
Best Regards,
Manuel
#9 - 21 Mar 2016 02:39 pm - Angelo Quaglia
From: Angelo Quaglia [mailto:[email protected]]Sent: 21 March 2016 14:38To: Illmeyer, Manuel ([email protected]) Subject: floating number of harvested metadata records
Dear Manuel,
Now that you that your account has, finally, been created in our Collaboration Environment, I would appreciate if we could exchange using theCollaboration Environment.
So, if you confirm you can update issues, this will be my last direct email to you.
This issue is ongoing since some time now and is being tracked at https://ies-svn.jrc.ec.europa.eu/issues/2509
Here is a chart that shows the variation in the number of the metadata records harvested from the Austrian National Discovery Service, since 1st
January 2016.
Harvestings are usually performed during the night between 6pm and 8am.
Apart from the number of metadata records, the Austrian National Discovery Service responds efficiently and reliably.
Do you know the reasons behind these fluctuations?
If necessary, we have the recordings of each harvesting session, including the GetRecordsReponse as they were received by the INSPIRE Geoportal.
If you need any additional information, I am of course available.
From: Illmeyer, Manuel [mailto:[email protected]]Sent: 24 March 2016 14:43To: Angelo Quaglia Subject: AW: floating number of harvested metadata records
Dear Angelo,
thanks i have received the invite. I will look into it soon.
Best regards,
Manuel
#11 - 30 Mar 2016 05:32 pm - Angelo Quaglia
Dear Manuel,
do you have any updates no this issue?
09 Jul 2018 13/25
Please update directly this issue using the tracker so that I do not have to copy and paste your emails here.
Best regards,
Angelo
#12 - 08 Apr 2016 10:50 am - Angelo Quaglia
From: Angelo Quaglia [mailto:[email protected]]Sent: 08 April 2016 10:49To: Illmeyer, Manuel ([email protected]) Cc: 'Jobst Markus' ; Melzer, Georg ; Seeger, Oliver Subject: (This problem was first reported to Austria on 06 Oct 2015): Floating number of harvested metadata recordsImportance: High
Dear Manuel,
I urge you to look into this issue that was reported to Austria on 5th October 2015 but has not been addressed yet.
Are you in a position to tell me what is the expected number of metadata records expected delivered by the Austrian National Discovery Service?
The number of metadata documents made available by the Austrian National Discovery Service keeps going up and down (see the diagram below):
03 Apr 2016, 23:25:01?Resources available for discovery: 911, Expected Resource Count: 911, Actual Resource Count : 911
08 Apr 2016, 03:47:44
Resources available for discovery: 901, Expected Resource Count: 901, Actual Resource Count : 901
901 is the current and correct amount of metadata records that are reported, respectively harvested, from the Austrian Discvovery Service.
Actually i do know, that there were some metadata, that have been depublished in the last few days and there have also been some changesregarding metadata in recent weeks.
To investigate this issue further we are, from now on, comparing the GetRecord responses from our discovery service, so that we can look into it, ifthere are massive changes again.
Above you mentioned, that you could give us the GetRecordsResponse of your Harvesting sessions. Could you send me some samples e.g. from:
I have uploaded here the file 20160408.zip which contains an excel file with the charts and also the original xml file produced by the geoportal. Wehave built a dashboard application but this has not been published, yet. In any case, Excel gives all the needed charting power.
I gather that the total records for Austria is therefore 901 from the National Discovery Service + 8 from the linked discovery services = 911 totalrecords from Austria.
Please note that Austria has chosen the Discovery Client scenario for the registration of the National Discovery Service with the INSPIRE Geoportal.
This means that the INSPIRE Geoportal recursively harvests each Discovery Service (also known as "Linked Discovery Service") described in eachmetadata harvested from the National Discovery Service.
09 Jul 2018 15/25
Therefore, a change in the toal number of metadata documents can be due to the National Discovery Service or to one of the linked ones.
In the validation reports the two figures are kept separate: the counts inside the green box refer to the National Discovery Service, while those in redboxes refer to the Linked Discovery Services:
Additional notes on the Discovery Client Scenario:
The list of services inside the FederateCatalogues element of the National Discovery Service capabilities document, if not empty, is only used to
09 Jul 2018 17/25
restrict the list of Discovery Services that are actually harvested.
If the list is empty, all Linked Discovery Services are harvested.
The Geoportal can actually detect if the same Discovery Service has already been harvested but the method it uses is not officially described in theTechnical Guidelines.
Therefore it respects the scenario description and assumes that different metadata documents describe different resources.
From the Discovery Service Guidelines:
Discovery client approach:
A combination of: the Publish Metadata operation (CSWT Manager.Transaction or CSWT Manager.Harvest) for publishingmetadata; CSW.GetCapabilities (“FederatedCatalogues”) forretrieving federated Discovery Service endpoints and/or CSW.GetRecords for discovering Discovery Service metadata; and CSW.GetRecordsto discover dataset or dataset series metadata from remote locations in a further step following the use of the here described link discovery service
4.3.4.2 Discovery client approach
The discovery client scenario is based on the availability of information on available Discovery Service endpoints in a Member State’s DiscoveryService.
Implementation Requirement 15
Third Party Discovery Services shall be published in the Member State’s Discovery Service using the Publish Metadata operation.Third Party Discovery Services can additionally be published in the “FederatedCatalogues” section of the Discovery Service’s capabilitydocument if they are part of a federated search infrastructure.
#15 - 18 Apr 2016 02:57 pm - Manuel Illmeyer
Hi Angelo,
thank you for the uploaded files an the explanation.
So you are saying that we can add the discovery services, which are anyway harvested by our national csw, to a list of "Federate Catalogues" so thatthey wont be harvested again by the jrc geoportal? I think this could solve the problem of the changing count of metadata elements shown by the jrcgeoportal.
In Austria (almost) all discovery services are harvested by our national catalogue. We only have problems with one csw cause of curled brackets insidede fileIdentifier element (these are the 8 records from the "INSPIRE Dienst via donau"). If we can solve the problems with them, we wouldn't need toharvest any service from austria, except the national discovery service.
Since we are monitoring the getRecords response from our csw we didn't discover any significant changes in the count of records. Only a rise from 901to 906. But this is expected and explainable due to some changes of a data provider.
Best regards,
09 Jul 2018 18/25
Manuel
#16 - 18 Apr 2016 03:06 pm - Manuel Illmeyer
*edit*
As I read the TG now I unterstand that only the services in the FederateCatalogues list are harvested? The "viadonau-csw" is already promoted there.
#17 - 18 Apr 2016 03:40 pm - Angelo Quaglia
In the Discovery.CLIENT scenario usually all Discovery Services for which metadata exists, are recursively harvested by the Geoportal.
However, if the FederatedCatalogues element is not empty, only the services listed there are harvested.
As I also wrote above:
Additional notes on the Discovery Client Scenario:
The list of services inside the FederateCatalogues element of the National Discovery Service capabilities document, if not empty, is only used torestrict the list of Discovery Services that are actually harvested.
If the list is empty, all Linked Discovery Services are harvested.
In any case, I have now an explanation for the bigger fluctuations: the FederatedCatalogues constraint sometimes "disappears" from thecapabilties of the National Discvoery Service of Austria.
The INSPIRE Geoportal records all responses as they are received from the external services.
The resource url: "http://inspire.via-donau.org/Geoportal/csw/discovery?request=GetCapabilities&service=CSW&version=2.0.2" did notrespond within 33000ms
#19 - 19 Apr 2016 12:47 pm - Angelo Quaglia
Dear Manuel,
the INSPIRE Geoportal cannot do much if case (1* ) occurs again.
The problem with case (2*) occurs when all services listed in the FederatedCatalogues constraint fail to respond.
In that case the list is empty and the geoportal harvests all linked Discovery Services.
The point is that the Geoportal needs to contact each linked Discovery Service to check it does not redirect to the main Discovery Service.
I have to think about the best algorithm for this specific case but it will probably be something like:
If errors occurred during the analisys of the FederatedCatalogues and the final list of linked Discovery Services is empty, do not harvest any of them.
We could find the problem for the 1* scenario. This was ozur backup discovery service. This service is running on a different infrastructure in adifferent organisation and serves as backup, if our service isn't available. We told them now to add the 1 service to the federated catalogues. So thisshouldn't happen again.
The 2* scenario is a bit of a problem to us, cause the discovery service listed in the federatedCatalogues section sometimes has maintenancedowntimes. In my opinion, if ther is a service entry in the federatedCatalogues section it shouldn#t harvest all CSW even if this one does not respond.
In fact I got an E-Mail today from the provider of the "INSPIRE CSW viaDonau" which told me that they have a downtime due to maintenance fromfriday (22.04.) evening to monday morning (25.04.). Is there anything we can do about this issue?
Thanks,
Manuel
#21 - 19 Apr 2016 07:10 pm - Angelo Quaglia
Dear Manuel,
many thanks for fixing issue 1.
Regarding issue 2, I agree with you in principle.
However, I must take into account that it is allowed and actually quite frequent for the federatedCatalogues section of a service, to contain theendpoint of the service itself.
In that case, the federatedCatalogues section is not empty but the geoportal is supposed to harvest from all linked services.
If the check failed, the geoportal would not harvest aby linked service.
It is unlikely for this to happen, but I do not want to take the risk because the bahaviour would not be easy to understand.
Please note that I cannot rely on syntactical comparison of URLs, rather I have to contact each endpoint to be sure of which service is actuallyresponding at the other end.
I think that the safest approach is the following one:
Discover CLIENT scenario
if
- the federated catalogues section is not empty
- no service is found (excluding the top service itself)
- at least one endpoint returned an error
then
09 Jul 2018 22/25
fail the harvesting operation
end if
This would mean that during the maintenance of "INSPIRE CSW viaDonau" the content of Austrian metadata would not get refreshed.
The only workaround would be to add a placeholder CSW endpoint that responds as a CSW but returns no records.
In this way the Geoportal would know that the intention is not to harvest all linked services.
#22 - 20 Apr 2016 08:59 am - Manuel Illmeyer
OK, thanks again.
I will discuss the option with the empty csw internally and write back to you as soon as possible.
#23 - 20 Apr 2016 04:26 pm - Angelo Quaglia
Another option is that I simply override the scenario from CLIENT to CENTRALISED during the maintenance days.
#24 - 25 Apr 2016 10:33 am - Manuel Illmeyer
Sorry, i was out of office.
The viadonau services should be working again. The long-term solution for Austria is the centralised scenario, but we have to fix our harvestingproblems first. We will switch to the newest geonetwork version in near future, which should solve the issue with the curled brackets.
#25 - 04 May 2016 09:23 am - Manuel Illmeyer
Until the implementation of the new geonetwork in Austria, I will inform you if there are maintenance days for the via Donau CSW again, so you canswitch the scenario during this time.
#26 - 04 May 2016 07:44 pm - Angelo Quaglia
OK, that's fine for me, many thanks.
09 Jul 2018 23/25
#27 - 23 Aug 2016 04:21 pm - Angelo Quaglia- Subject changed from Austria - Land-, forst- und wasserwirtschaftliches Rechenzentrum GmbH: Observed reduction in the number of metadata records to Austria - LFRZ: Observed reduction in the number of metadata records- Submitting Organisation changed from Austria - Land-, forst- und wasserwirtschaftliches Rechenzentrum GmbH to Austria - LFRZ
#28 - 23 Aug 2016 04:22 pm - Angelo Quaglia- Subject changed from Austria - LFRZ: Observed reduction in the number of metadata records to AT - LFRZ: Observed reduction in the number of metadata records- Submitting Organisation changed from Austria - LFRZ to AT - LFRZ
#29 - 24 Aug 2016 06:24 pm - Angelo Quaglia- Category set to Harvesting results
#30 - 21 Sep 2016 06:26 pm - Angelo Quaglia- Country set to AT - Austria
#31 - 13 Apr 2017 10:26 am - Angelo Quaglia- Proactive set to No
Dear Manuel,
the service https://geometadaten.lfrz.at/at.lfrz.discoveryservices/srv/de/csw202declares 926 records:Resources available for discovery: 926, Expected Resource Count: 926, Actual Resource Count : 926
Yesterday, 928:Result of the interaction with the Discovery ServiceResources available for discovery: 928, Expected Resource Count: 928, Actual Resource Count : 928
Could you please tell me what is the expected number of records?
Best regards,
Angelo
#32 - 13 Apr 2017 10:27 am - Angelo Quaglia- Proactive changed from No to Yes
