×By the end of May 2018, the ticketing system at https://ticket.opensciencegrid.org will be retired and support will be provided at https://support.opensciencegrid.org. Throughout this transition the support email (help@opensciencegrid.org) will be available as a point of contact.

Please see the service migration page for details: https://opensciencegrid.github.io/technology/policy/service-migrations-spring-2018/#ticket
Serguei Fedorov
GOC Ticket/submit
Serguei Fedorov
Mat to debug
Scott Teige / OSG Operations Infrastructure
Dave Dykstra / OSG Security Coordinators
Software Support (Triage) / OSG Software Team
Matyas Selmeci / OSG Software Team
Tim Theisen / OSG Software Team

Assignees TODO
Past Updates

If there is any problems with certificates in cvmfs repository again  ?

voms-proxy-init -voms cms -valid 192:00 -debug



Detected Globus version: 2.2
Unspecified proxy version, settling on Globus version: 2
Number of bits in key :1024
Files being used:
CA certificate file: none
Trusted certificates directory : /cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.2/3.2.26/el6-x86_64/etc/grid-security/certificates
Proxy certificate file : /tmp/x509up_u0
User certificate file: /etc/grid-security/hostcert.pem
User key file: /etc/grid-security/hostkey.pem
Output to /tmp/x509up_u0
Your identity: /DC=com/DC=DigiCert-Grid/O=Open Science Grid/OU=Services/CN=earth.crc.nd.edu
Using configuration file /cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/3.2/3.2.26/el6-x86_64/etc/vomses
Creating temporary proxy to /tmp/tmp_x509up_u0_29211 ...++++++
Contacting  voms2.cern.ch:15002 [/DC=ch/DC=cern/OU=computers/CN=voms2.cern.ch] "cms" Failed

Error: Error during SSL handshake:error:80066405:lib(128):verify_callback:outdated CRL found, revoking all certs till you get new CRL:sslutils.c:2115
outdated CRL found, revoking all certs till you get new CRL
Function: verify_callback
error:80066411:lib(128):verify_callback:certificate failed verify::sslutils.c:2318
error =CRL has expired
issuer =/DC=ch/DC=cern/CN=CERN Grid Certification Authority
by Kyle Gross 
I added Tim T to this ticket since he did the last update to the repo as far as I can see.  He may know more.

By the way, is it still recommended to use the OSG CVMFS certificates?


or is it better to use osg-ca-certs-updater-cron + fetch-crl daemons to keep the local ones up to date?
What are the advantages/disanvantages for each option?


by /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=khurtado/CN=764581/CN=Kenyi Paolo Hurtado Anampa
by Tim Theisen 
Mat was the last person to deal with this. He knows the current status.
by Dave Dykstra 
The mis vo hasn't done an osg-oasis-update since Monday.

On oasis-login there's a process that's been stuck since then running under the ouser.mis login:
python2 /home/ouser.mis/oasis-ca-updater
I don't have root access on oasis-login so I don't know what it is stuck on.  It doesn't have any subprocesses.

Adding Scott as the administrator of oasis-login.
by Scott Teige 
Killed the offending process, ran an osg-oasis-update and oasis-ca-updater as ouser.mis, all completed without intervention.
Will observe next scheduled run in 43 minutes.
by Scott Teige 
next update runs at 16:20, will check then. /cvmfs/oasis.opensciencegrid.org/mis/certificates shows content timestamped at the time of my forced update. Everything worked manually will wait for 16:20 to verify automated update completed properly.
by Rob Quick 
I now see the updated CA package when accessing OASIS from xd-login. This should be fixed. We'll do an analysis of the problem on Tuesday.
by Scott Teige 
I ran the updater manually through the weekend. It seems to be leaving behind a lockfile, handing this off to Tim T. Ops is, at this time, uninvolved.
De-asssigning Kyle,  I'll lurk in case I need to continue manual intervention.

We would like to know what source for CRLs is recommended to use  ?
I was told before that cvmfs is reliable but since I experienced the problems I switched to local.

Do I have to switch back to cvmfs repository ?



by /DC=com/DC=DigiCert-Grid/O=Open Science Grid/OU=People/CN=Serguei Fedorov 787
by Matyas Selmeci 
The existence of the lockfile is perfectly normal -- the script uses flock to actually lock/unlock the file, instead of just checking for its existence. More worrying is that the script kept running - it's set to time out after 60 minutes. I'll crank up the debug level and see what I can find.

by Scott Teige 
fetch-crl is supported, something else might be supported at some future date, this will be discussed soon.
by Scott Teige 
Hello, access via cvmfs is now monitored, if the CRLs are not updated in 24 hrs, the service will go into a warning state, 48 hrs will go into a critical state. A problem will be addressed the next business day. Please note, this addresses updating of the CRLs. The CRLs are accessible via three redundant cvmfs replica located at IU, BNL and FNAL. It is extremely unlikely all three will fail simultaneously but such an event is considered of critical priority.

There is no policy, at this time, stating which method is referred. I hope the above information is sufficient for you to choose which you prefer and understand there may be external constraints forcing you to choose the /cvmfs option.

I'm resolving this ticket since the original problem has been addressed. Mat, if you'd like, create and track your work with a JIRA ticket. As always, if you encounter problems, please let us know as soon as possible and we will assist.
Similar Recent Tickets modified within the last 30 days

No similar tickets found.

GOC Ticket Version 2.2 | Report Bugs | Privacy Policy

Copyright 2018 The Trustees of Indiana University - Developed for Open Science Grid