Perun external sources

Z MetaCentrum
Skočit na navigaci Skočit na vyhledávání

External sources are used to get information about users from external systems. Perun has several predefined connectors: LDAP and SQL. External source defines how to connect to the external system (url, hostname, login, password, ...) and what are the transformation rules between external system and Perun.

Perun external sources definition

Already implemented:

  • cz.metacentrum.perun.core.impl.ExtSourceXML
  • cz.metacentrum.perun.core.impl.ExtSourceLdap
  • cz.metacentrum.perun.core.impl.ExtSourceSql
  • cz.metacentrum.perun.core.impl.ExtSourceISMU
  • cz.metacentrum.perun.core.impl.ExtSourceCSV
  • cz.metacentrum.perun.core.impl.ExtSourceGoogle
  • cz.metacentrum.perun.core.impl.ExtSourceREMS

After a change of the configuration file with external resources "/etc/perunv3/perun-extSources.xml", command"./loadExtSourcesDefinitions" must be run to reload configuration file.

XML

IMPORTANT: In queries part (?) can't be used character ['] cause problem of escaping (for now) 

 Type: cz.metacentrum.perun.core.impl.ExtSourceXML

Connection parameters (for local file):

  • file - path to file with xml (local) e.g. /etc/perun/file.xml (if file is filled, use file)

Connection parameters (for external XML file from web aplication):

  • uri - uri where xml exists e.g. https://something.cz/getListOfMembers (if file is not filled, uri must be filled and then is used)
  • keyStore - location of keystore with certificate e.g. /etc/perun/perun.p12 (for ssl comunicaton)
  • keyStorePass - password for keystore certificate e.g. password (for ssl comunicaton)
  • keyStoreType - type of keystore certificate e.g. PKCS12 (for ssl comunicaton, PKCS12 is recomended)
  • trustStore - trust store location with certificates of server e.g. /etc/perun/truststore (for ssl comunicaton)
  • trustStorePass - password to trustStore certificates e.g. password (for ssl comunicaton)

Other parameters:

  • xpath - xpath query where ? will be assigned required value e.g. //member[starts-with(DN/text(),'?')
  • loginXpath - xpath query returning particular record according to the user login. It must return one or no record, e.g. //member[DN/text()='?']
  • xmlMapping - pairing of attribute name in Perun and xpath query, e.g. login=./DN/text(). As a value must be local xpath query or regex with xpathQuery
    • local query - ./DN/text()
    • xpath + regex - #[regex]/[replacement]#[xpath query] e.g. #^.*(CN|cn)([abc]+)$/$2#./DN/text()
    • when we want to add another userExtSources to user, we use following parameter. First add more userExtSources, then add numbering. Syntax: additionalues_[no.]=concat([ExtSource name],'|[ExtSource type]|', [Pattern for extLogin], '|[LoA]')

UserExtSource example: 

additionalues_1=concat(./CA/text(),'|cz.metacentrum.perun.core.impl.ExtSourceX509|', ./DN/text(), '|2' )

XML external source example: 

 <extSource>
   <name>BIOMEDVOMS</name>
   <type>cz.metacentrum.perun.core.impl.ExtSourceXML</type>
   <description></description>

   <attributes>
     <attribute name="keyStore">/ExtSourceVoms/perun-cesnet-cz.p12</attribute>
     <attribute name="keyStorePass">pass</attribute>
     <attribute name="keyStoreType">PKCS12</attribute>
     <attribute name="trustStore">/ExtSourceVoms/trustStore</attribute>
     <attribute name="trustStorePass">pass</attribute>
     <attribute name="uri">https://cclcgvomsl.cz/listMembers</attribute>
     <attribute name="file"></attribute>
     <attribute name="xpath">//multiRef[starts-with(DN/text(),'/') and contains(DN/text(),'?')]</attribute>
     <attribute name="loginXpath">//multiRef[starts-with(DN/text(),'/') and DN/text()='?']</attribute>
     <attribute name="xmlMapping">
       additionalues_1=concat(./CA/text(),'|cz.metacentrum.perun.core.impl.ExtSourceX509|', ./DN/text(), '|2' ),
       login=./DN/text(),
       urn:perun:member:attribute-def:def:mail=./mail/text(),
       firstName=#^.*(CN|cn)=([0-9]*\s*)(([a-zA-Z]*)[- .])?([-a-zA-Z. ']*)([ ,\/].*)?$/$4#./DN/text(),
       lastName=#^.*(CN|cn)=([0-9]*\s*)(([a-zA-Z]*)[- .])?([-a-zA-Z. ']*)([ ,\/].*)?$/$5#./DN/text()
     </attribute>
   </attributes>
 </extSource>

LDAP

 Type: cz.metacentrum.perun.core.impl.ExtSourceLdap

Connection parameters:

  • url - e.g. ldap://meta-ldap.cesnet.cz:389/ou=People,o=meta,c=cz
  • user - e.g. CN=test,OU=Users,O=perun
  • password
  • referral - e.g. follow
  • query - query template to LDAP, as ? will be assigned required value e.g. {{{(uid=*?*)}}} or {{{(CN=?)}}}
  • loginQuery - query template to LDAP, returning particular record according to the user login. It must return one or no record, e.g. {{{(CN=?}}}
  • ldapMapping - pairing of attribute name in Perun and item name in LDAP, e.g. firstName={givenName}. Item name in LDAP must be closed into {}. As a value could be used constant or a combination with LDAP item, for example login={uid}@muni.cz
    • when attribute in LDAP contains one or more value but only one is wanted, use index. For example: {mail[0]}, take the first of mails.
    • when attribute in LDAP contains more values, that must be mapped to more attributes in Perun, regular expressions can be used:[perunAttrName]={[ldapAttrName]|[regex]}, for example: attribute "title" in LDAP contains "RNDr.:PhD." into ldapMapping we put two definitions
{{{ titleBefore={title|^(.*):.*},titleAfter={title|^.*:(.*)} }}}

Value in "()" in regexp will be taken to each definition.

    • Each item in ldapMapping must be on new-line separated by comma, see example below.
    • When we want to add another userExtSources to user, we use following parameter. First add more userExtSources, then add numbering. Syntax:
additionalues_[no.]=[ExtSource name]|[ExtSource type]|[Pattern for extLogin]|[LoA]

UserExtSource example: 

additionalues_1=https://idp2.ics.muni.cz/idp/shibboleth%7Ccz.metacentrum.perun.core.impl.ExtSourceIdp%7C{cn}@muni.cz
additionalues_2=IS.MUNI.CZ|cz.metacentrum.perun.core.impl.ExtSourceKerberos|{cn}@IS.MUNI.CZ

LDAP external source example: 

 <extSource>
   <name>LDAPMU</name>
   <type>cz.metacentrum.perun.core.impl.ExtSourceLdap</type>
   <description></description>
   <attributes>
     <attribute name="url">ldap://server</attribute>
     <attribute name="base">OU=Users,OU=Org,DC=us</attribute>
     <attribute name="query">(CN=?)</attribute>
     <attribute name="loginQuery">(CN=?)</attribute>
     <attribute name="user">CN=admin,OU=System,DC=us</attribute>
     <attribute name="password">bla</attribute>
     <attribute name="referral">follow</attribute>
     <attribute name="ldapMapping">
       firstName={givenName},
       lastName={sn},
       login={cn},
       urn:perun:member:attribute-def:def:mail={mail[0]},
       urn:perun:member:attribute-def:def:organization=My organization,
       urn:perun:user:attribute-def:def:login-namespace:mu={cn},
       titleBefore={title|^(.*):.*},
       titleAfter={title|^.*:(.*)},
       additionalues_1=https://idp.org.us/idp/shibboleth%7Ccz.metacentrum.perun.core.impl.ExtSourceIdp%7C{cn}@org.us%7C2
     </attribute>
   </attributes>
 </extSource>

SQL

 Type: cz.metacentrum.perun.core.impl.ExtSourceSql

Connection parameters:

  • driver - selecting DB driver e.g. com.mysql.jdbc.Driver
  • url - JDBC url, e.g.. jdbc:mysql://localhost:3307/DB
  • user
  • password
  • query - This query is used to find person record in external DB, it can be used for fulltext search. Column name must match with attribute definition in Perun. The '?' is then replaced by the search string, e.g. when you are searching user while adding into the VO.

Because some of the DB engines has limitation on column name length, attribute names have to be shorted by following way - x:y:z :

x means
m member
u user
f facility
r resource
g group
v vo
h host
mr member_resource
uf user_facility
gr group_resource


y means
d def
o opt


z - means value of attribute

Examples of shorting of attributes: 

m:d:address -> urn:perun:member:attribute-def:def:address
u:d:login-namespace:mu -> urn:perun:user:attribute-def:def:login-namespace:m

For attributes firstName, lastName, middleName, titleBefore, titleAfter and login, you do not need to specify attribute name, these are predefined.

For example: 

{{{  select name as first_name, surname as last_name, login, mail as "m:d:mail", telephone as "m:d:phone", logname as login from someTable where (first_name || ' ' || last_name) like '%' || ? || '%' }}}

Will do the fulltext search for the user, matching [first_name last_name].

  • loginQuery - This query is used to get information about particular person, login returned by 'query' will be used as a replacement for '?' in loginQuery. The loginQuery must return one or no record.

For example 

{{{ select first_name as firstName, last_name as lastName, ''   as middleName, title as titleBefore, ''  as titleAfter, mail as "m:d:mail", logname as login from someTable where logname=? }}}
    • When we want to assign another userExtSources to user, we use following syntax

Syntax: 

[ExtSource name]|[ExtSource type]|[Pattern for extLogin]|[LoA] as additionalues_[no.]

For example, just add it to the select query (MySQL example): 

concat(ca.ca,'|cz.metacentrum.perun.core.impl.ExtSourceX509|',usr.dn,'|',2) as `additionalues_1
concat("EINFRA",'|cz.metacentrum.perun.core.impl.ExtSourceKereros|',usr.login,'|',2) as `additionalues_2

Example of SQL external source (MySQL): 

 <extSource>
   <name>BLA</name>
   <type>cz.metacentrum.perun.core.impl.ExtSourceSql</type>
   <description></description>
   <attributes>
     <attribute name="driver">com.mysql.jdbc.Driver</attribute>
     <attribute name="url">jdbc:mysql://localhost:3307/DB</attribute>
     <attribute name="query">
       select
        usr.userid as `login`,
        SUBSTRING_INDEX(cn, ' ',1) as `firstName`,
        (case SUBSTRING(cn, length(SUBSTRING_INDEX(cn, ' ',1))+2) when  then 'N/A' else
        SUBSTRING(cn, length(SUBSTRING_INDEX(cn, ' ',1))+2) end) as `lastName`,
        usr.mail as `m:d:mail`,
        usr.mail as `u:d:preferredMail`,
        concat(ca.ca,'|cz.metacentrum.perun.core.impl.ExtSourceX509|',usr.dn,'|',2) as `additionalues_1`
       from
        usr, ca
       where
        usr.userid=? and
        usr.ca=ca.cid
     </attribute>
     <attribute name="loginQuery">
       select
        usr.userid as `login`,
        SUBSTRING_INDEX(cn, ' ',1) as `firstName`,
        (case SUBSTRING(cn, length(SUBSTRING_INDEX(cn, ' ',1))+2) when  then 'N/A' else
        SUBSTRING(cn, length(SUBSTRING_INDEX(cn, ' ',1))+2) end) as `lastName`,
        usr.mail as `m:d:mail`,
        usr.mail as `u:d:preferredMail`,
        concat(ca.ca,'|cz.metacentrum.perun.core.impl.ExtSourceX509|',usr.dn,'|',2) as `additionalues_1`
       from
        usr, ca
       where
        usr.userid=? and
        usr.ca=ca.cid
     </attribute>
     <attribute name="user">admin</attribute>
     <attribute name="password">admin</attribute>
   </attributes>
 </extSource>

IS MU

 Type: cz.metacentrum.perun.core.impl.ExtSourceISMU

Used for synchronization with IS Masaryk University.


Connection parameters:

CSV

 Type: cz.metacentrum.perun.core.impl.ExtSourceCSV

Connection parameters:

  • file - path to CSV file (local) e.g. /etc/perun/file.csv, in this case CSV file has to contain header on the first row
  • query - query for finding users (rows in CSV file) with given attributes, e.g. eppn contains ?, where attribute eppn represents column name from CSV header and contains is a key word that is mandatory for query parameter
  • loginQuery - query for finding a user (1 row in CSV file) with given primary attribute, e.g. login=?, where attribute eppn represents column name from CSV header
  • csvMapping - pairing of CSV header (column) names to their Perun attributes e.g. urn:perun:member:attribute-def:def:mail or static constant e.g. login.


CSV file example: 

login,eppn,email,first_name,last_name
xuser1,123456@cesnet.cz,xuser1@mail.muni.cz,"Uživatel 1","Příjmení"
xuser2,123457@cesnet.cz,xuser2@mail.muni.cz,"Uživatel 2","Příjmení"
xuser3,123458@cesnet.cz,xuser3@mail.muni.cz,"Uživatel 3","Příjmení"
xuser4,123459@cesnet.cz,xuser4@mail.muni.cz,"Uživatel 4","Příjmení"

CSV external source example: 

 <extSource>
   <name>CSV</name>
   <type>cz.metacentrum.perun.core.impl.ExtSourceCSV</type>
   <description>Testing ext source for CSV files</description>
   <attributes>
     <attribute name="file">/home/perun/members.csv</attribute>
     <!-- When you search for all possible users by "searchString" -->
     <attribute name="query">eppn contains ?</attribute>
     <!-- When you ask for single user by login -->
     <attribute name="loginQuery">login=?</attribute>
     <!-- mapping of CSV column name to perun attribute or known constant -->
     <attribute name="csvMapping">
       first_name={firstName},
       last_name={lastName},
       login={login},
       eppn={eppn},
       email={urn:perun:member:attribute-def:def:mail}
     </attribute>
   </attributes>
 </extSource>

Google

 Type: cz.metacentrum.perun.core.impl.ExtSourceGoogle

Connection parameters:

  • domain - domain url, where the users come from
  • group - Google group name in domain, from which the users are imported to Perun. It has to contain whole address of the group, that means something like this: name1@einfra.cesnet.cz (including '@domainName)
  • query - query for finding users in given Google group with given attributes, e.g. email contains ?, where contains is a key word that is mandatory for query parameter. It is possible to query only 'email' parameter
  • loginQuery - query for finding a user in given Google group with given primary attribute, e.g. id=?. It is possible to query only 'email' and 'id' parameters.
  • serviceAccountEmail - an email generated according to this guide.
  • userEmail - an email address of the user in the domain. Application will act like this user and will query for users in specified group.
  • p12File - path to the file containing P12 key generated according to guide mentioned above
  • googleMapping - pairing of attribute name in Perun and value from Google group, e.g. urn:perun:user:attribute-def:virt:logins-namespace:google={userID}. Only {userID}, {domainName} and {groupName} can be connected like this, all other attributes in Perun have to be connected with exact value, e.g. urn:perun:user:attribute-def:def:preferredMail=name@gmail.com.

Google external source example: 

 <extSource>
   <name>GOOGLE</name>
   <type>cz.metacentrum.perun.core.impl.ExtSourceGoogle</type>
   <description></description>
   
   <attributes>
     <attribute name="domain">einfra.cesnet.cz</attribute>
     <attribute name="group">name1@einfra.cesnet.cz</attribute>
     <attribute name="query">email contains ?</attribute>
     <attribute name="loginQuery">id=?</attribute>
     <attribute name="serviceAccountEmail">id@developer.gserviceaccount.com</attribute>
     <attribute name="userEmail">login@einfra.cesnet.cz</attribute>
     <attribute name="p12File">/etc/perun/client_secret_pkcs12.p12</attribute>
     <attribute name="googleMapping">
       urn:perun:user:attribute-def:virt:logins-namespace:google={userID},
       urn:perun:facility:attribute-def:def:googleGroupNameNamespace={domainName},
       urn:perun:group:attribute-def:def:googleGroupName-namespace:einfra.cesnet.cz={groupName},
       urn:perun:user:attribute-def:def:preferredMail=name@gmail.com
     </attribute>
   </attributes>
 </extSource>


REMS

 Type: cz.metacentrum.perun.core.impl.ExtSourceREMS

Connection parameters:

  • driver - selecting DB driver e.g. com.mysql.jdbc.Driver
  • url - JDBC url, e.g.. jdbc:mysql://localhost:3307/DB
  • user
  • password
  • query - This query is used to find person record in external DB, it can be used for fulltext search. Column name must match with attribute definition in Perun. The '?' is then replaced by the search string, e.g. when you are searching user while adding into the VO.

This external source works the same way as ExtSourceSql except one thing:

  • It returns only those subjects that are already in Perun. That is decided by two aspects:
    • In perun exists an User with ExtSourceREMS with the same login.
    • In perun exists an ExtSource of given type and value. This must be returned in Sql query and it should have the following format:

`{extSourceName}|{extSourceClass}|{eppn}|0` and the name of this value must starts with `additionalues_`.

Example of REMS external source (MySQL): 

 <extSource>
   <name>BLA</name>
   <type>cz.metacentrum.perun.core.impl.ExtSourceREMS</type>
   <description></description>
   <attributes>
     <attribute name="driver">com.mysql.jdbc.Driver</attribute>
     <attribute name="url">jdbc:mysql://localhost:3307/DB</attribute>
     <attribute name="query">
       select
        id as `login`,
        (CASE 
          WHEN resource like 'BonaFideStatus' THEN "http://www.ga4gh.org/beacon/bonafide/ver1.0"
          ELSE null)
         END CASE) AS "u:d:elixirBonaFideStatusREMS",
        entityId || '|cz.metacentrum.perun.core.impl.ExtSourceLdap|' || eppn || '|0’ as `additionalues_1`
       from
        someTable
     </attribute>
     <attribute name="loginQuery">
       select
        id as `login`,
        (CASE 
          WHEN resource like 'BonaFideStatus' THEN "http://www.ga4gh.org/beacon/bonafide/ver1.0"
          ELSE null)
         END CASE) AS "u:d:elixirBonaFideStatusREMS",
        entityId || '|cz.metacentrum.perun.core.impl.ExtSourceLdap|' || eppn || '|0’ as `additionalues_1`
       from
        someTable
     </attribute>
     <attribute name="user">admin</attribute>
     <attribute name="password">admin</attribute>
   </attributes>
 </extSource>
Citováno z „https://wiki.metacentrum.cz/w/index.php?title=Perun_external_sources&oldid=7515“