Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resume the harvesting processing when the oai pmh server is not responding #67

Open
Aazhar opened this issue Dec 14, 2016 · 2 comments
Open
Assignees

Comments

@Aazhar
Copy link
Member

Aazhar commented Dec 14, 2016

Use the last diagnosis like error 500 on the oai-pmh server.

@Aazhar Aazhar self-assigned this Dec 14, 2016
@lfoppiano
Copy link
Member

I'm having some problems while downloading, the process terminate without any error message.

e.g. this was the message I've got and the java process was not running anymore.

04 Jan 2017 10:40.02 [INFO ] OAIPMHHarvester           - 	 Sending: http://api.archives-ouvertes.fr/oai/hal/?verb=ListRecords&metadataPrefix=xml-tei&from=2002-03-09&until=2002-03-09

On a side note, while investigating I've discovered that here you assume that an exception means having no binary file. I think it's an assumption a bit too strong because any error that is not managed in the Utilities.getconnection()'s method will result in the wrong status.

@lfoppiano
Copy link
Member

Here another example from today (15:10 - Sat 7Jan):

[lfoppian@traces3 harvest]$ tail harvest.log
07 Jan 2017 04:00.01 [INFO ] OAIPMHHarvester           - 		 Processing TEI for halshs-00412216
07 Jan 2017 04:00.01 [INFO ] OAIPMHHarvester           - 				 Storing TEI halshs-00412216
07 Jan 2017 04:00.01 [INFO ] OAIPMHHarvester           - 				 PDF not found !
07 Jan 2017 04:00.01 [INFO ] OAIPMHHarvester           - 		 Processing TEI for halshs-00412217
07 Jan 2017 04:00.01 [INFO ] OAIPMHHarvester           - 				 Storing TEI halshs-00412217
07 Jan 2017 04:00.01 [INFO ] OAIPMHHarvester           - 				 PDF not found !
07 Jan 2017 04:00.01 [INFO ] OAIPMHHarvester           - 		 Processing TEI for halshs-00412218
07 Jan 2017 04:00.01 [INFO ] OAIPMHHarvester           - 				 Storing TEI halshs-00412218
07 Jan 2017 04:00.01 [INFO ] OAIPMHHarvester           - 				 PDF not found !
07 Jan 2017 04:00.01 [INFO ] OAIPMHHarvester           - 	 Sending: http://api.archives-ouvertes.fr/oai/hal/?verb=ListRecords&resumptionToken=
[lfoppian@traces3 harvest]$ ps afx  | grep harvest
33226 pts/0    S+     0:00              \_ grep --color=auto harvest
[lfoppian@traces3 harvest]$ 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants