# Welcome to the Public Access Submission System (PASS) Documentation Welcome to the Public Access Submission System (PASS) Documentation! If you're a first-time visitor, we recommend beginning with our Welcome Guide, accessible that you start with the welcome guide [here](/welcome-guide). This guide offers a comprehensive introduction to PASS, detailing the unique challenges it addresses for researchers, guiding you through the initial setup, and providing a detailed overview of the PASS architecture. Dive in to discover how PASS simplifies and streamlines the submission process for your research! If you're already familiar with PASS and want jump right in, you can find our developer docs [here](/developer-documentation) or our infrastructure docs [here](/infrastructure-documenation) # PASS Welcome Guide Welcome to the Public Access Submission System (PASS) documentation! The Public Access Submission System (PASS) is an Eclipse open source platform designed to assist researchers, IT staff, compliance officers, and executives in efficiently and economically complying with the access policies of their funders and institutions. Use Eclipse PASS to submit your manuscripts to funder and institutional publication repositories (e.g. PubMed Central, DSpace) and comply with access policies. Using the web-based PASS system, you can avoid paying article processing charges to make your publication open to the public, simultaneously send your manuscript to multiple repositories seamlessly, populate forms automatically with publication and author information by providing DOIs and ORCID IDs. In this guide we step through various topics on PASS: * Research submissions overview * Problems Researchers face when submitting to different repositories * How PASS is solving submission challenges for researchers * PASS at JHU * PASS demonstrations at Conferences * Technology stack * Deployment architecture * Latest release * Setup and run PASS * Collaboration with other institutions * Contributing to PASS # Research Submission Overview The journey of research from conception to dissemination is a long one. After designing your study, gathering and analyzing your data, undergoing peer review, there is still more to be done. For many researchers the next stage is submitting your manuscripts and all the supporting documents to a variety of repositories, as dictated by the requirements of funding bodies and academic institutions. ## Challenges Faced by Researchers Identifying the appropriate repositories for submission is not always straightforward. Researchers often juggle multiple grants, each with its unique set of requirements. Gathering all this information can be time consuming and using many systems can be tedious and frustrating. The workflows for each system can be complex and navigating these systems can be burdensome. Additionally, there can be processing charges for each repository, which are often incurred on a per-repository basis, cannot be overlooked. ## PASS: Solving submission challenges The Eclipse Public Access Submission System (PASS) is a web-based system that aims to solves these challenges. PASS is engineered to support researchers, IT personnel, compliance officers, and executives in meeting the access policies of their funders and institutions both efficiently and economically. By leveraging Eclipse PASS to submit your manuscripts to both funder and institutional publication repositories (such as PubMed Central, DSpace) your workflow becomes streamlined, ensuring compliance and avoiding costly article processing charges to make your publication open to the public. By simply signing into PASS, it gathers all the relevant information about your grants, publications, and previous submissions to populate forms with publication and author information by providing DOIs and ORCID IDs. It simultaneously sends your manuscript to multiple repositories seamlessly with a few single clicks! Our mission is to support researchers, and we know the frustrations and challenges researchers face trying to get their research easily accessible in the public domain. PASS was originally built in collaboration by the Eclipse Foundation and the Digital Research Curation Center at Johns Hopkins Sheridan Libraries. As an evolving open-source project, PASS welcomes the collaboration of universities and institutions, aiming to continually refine and enhance a platform that frees up researchers to do what they're passionate about: pushing the boundaries of what we know. # PASS at JHU Since its inception in 2018, PASS at Johns Hopkins University (JHU) has been a vehicle for research submission and compliance. In 2023, we proudly introduced our 1.0 release, featuring a major overhaul of our back-end architecture. This update introduced a robust and well-defined API, streamlined our release process, aligned dependencies, and restructured components for easier system management. Designed to fit the needs of the Office of Science and Technology [memo ](https://www.whitehouse.gov/wp-content/uploads/2022/08/08-2022-OSTP-Public-Access-Memo.pdf)and aligns with JHU’s own Open Access Policy, PASS ensures that all JHU researchers can effortlessly publish their work to JHU's institutional repository, [JScholarship](https://provost.jhu.edu/faqs/what-is-jscholarship/). Beyond meeting JHU’s requirements, PASS at JHU facilitates submissions to PubMed Central through the [NIH Manuscript Submission](https://www.nihms.nih.gov/) (NIHMS) system. Looking ahead, we’re excited to expand our support to additional repositories, such as NSF. Take a look at our [PASS Roadmap](https://github.com/eclipse-pass/pass-documentation/blob/main/welcome-guide/pass-roadmap.md) article for a glimpse into the future features and enhancements in the pipeline. Architecturally, PASS is built on Amazon Web Services and leverages a Java Spring Boot web application paired with a REST API for seamless core service integration. While we’ve chosen AWS for its reliability and scalability, it’s essential to note that PASS’s flexibility allows it to thrive on various platforms. For an in-depth look at our system’s blueprint, we invite you to explore our [Deployment Architecture](/welcome-guide/deployment-architecture) article. # PASS Demonstrations at Conferences ### Eclipse Pass Project Briefing, January 2024 In 2023, the PASS team at Johns Hopkins University achieved significant milestones including a series of monthly releases that introduced a new back-end architecture, enhanced API, and a range of system improvements to support a more efficient and user-friendly experience. Alongside development, the team engaged in community building with academic partners and hosting providers to explore piloting opportunities and integration needs for 2024. The PASS project plans to focus on expanding user engagement, building a broader community of institutions interested in utilizing PASS, enhancing system administration tools, and making application improvements, including better integration with funder repositories. For more details visit the [Project Briefing](https://drive.google.com/file/d/1WvAUQLLXbGAsCjBDgjUK-so0glmjwsaF/view) *** ### CNI Presentation, December 2023 {% embed url="" %} PASS CNI Presentation December 2023 {% endembed %} For more details about this presentation visit the [CNI website](https://www.cni.org/topics/ci/the-nsf-public-access-initiative-projects-funded-and-catalytic-aims-of-the-program) *** ### Eclipse PASS Project Briefing, January 2023 In 2022, the Public Access Submission System (PASS) project embarked on significant advancements, notably transitioning to an Eclipse Foundation open-source project, and upgrading its server architecture. This strategic move to the Eclipse Foundation enhanced PASS with engineering, community management, and organizational governance expertise, enabling more effective collaboration with agencies and institutions. Furthermore, the architectural upgrade enhanced the system's API, simplified deployment, and streamlined system management, setting the stage for broader community engagement and institutional collaboration. For more details visit the [Project Briefing](https://drive.google.com/file/d/12jCpURDYDbfiAnzjBMeukL1zSBsf-hB8/view) *** ### CNI Presentation, December 2022 {% embed url="" %} PASS CNI Presentation December 2022 {% endembed %} View the [Presentation Slides](https://www.cni.org/wp-content/uploads/2022/12/Eclipse-PASS-CNI-Presentation-2022-12-13-Bill-Branan.pdf), or more details about this presentation visit the [CNI Website](https://www.cni.org/topics/ci/how-the-public-access-submission-system-is-ideally-suited-to-address-the-new-ostp-memorandum) *** ### CNI Presentation, Spring 2020 {% embed url="" %} PASS CNI Presentation Spring 2020 {% endembed %} For more details about this presentation visit the [CNI Website](https://www.cni.org/topics/repositories/packaging-specification-for-simultaneous-deposit-of-articles-and-data-into-multiple-repositories) *** ### FORCE11 Presentation, November 2018 {% embed url="" %} PASS FORCE11 Presentation November 2018 {% endembed %} View the [Presentation Slides](https://zenodo.org/records/1453344), or for more details about the presentation visit the [FORCE11 Website](https://force11.org/force2018/) # Technology Stack ### Introduction PASS is composed of four main components: PASS UI, Data Loaders, PASS Core, and Deposit Services. These components are integral to the application running, but are flexible enough that they can be stood up on their own. Built with an array of open-source technologies, PASS guarantees a sturdy, secure, and transparent framework for deployment across diverse environments. PASS UI contains the Submission UI which is responsible for the researcher workflow to review their grants and create submissions to their designated repositories, all while adhering to the policies established by the PASS Core's policy engine. PASS UI interfaces with PASS Core through Shibboleth authentication to support a single sign-on (SSO) experience, facilitating secure and streamlined access to the system's core functionalities. PASS Core is the central back-bone to the application which is responsible for orchestrating the entire researcher workflow by communicating with external services, managing the data layer, applying policy and metadata rules. Data pertaining to journals, grants, and publications is funneled into PASS Core via Data Loaders, where it is then managed by the Java Persistence API (JPA). PASS Core also employs the Oxford Common File Layout (OCFL) for the storage of submission-related documents, offering options for disk or Amazon S3 bucket retention. When researchers are performing their submissions, it will flow through the PASS API. From there, a submission message is placed in a publication queue which the deposit services will pick up, assemble the deposit and transport it to their respective repository (DSpace, PubMed, etc).

PASS Architecture

### Front-end Technologies To support the Submission UI the following technologies and frameworks are employed by PASS: * [Ember.js](https://emberjs.com/): Selected for its robustness and opinionated framework structure, the Submission UI uses Ember.js to provide a clear, consistent, and easy to use workflow. The UI is written in TypeScript using Glimmer Template Syntax (GTS) components. * [Vite](https://vite.dev/): The build tool and development server for pass-ui, providing fast hot module replacement (HMR) during development and optimized production builds. * [WarpDrive](https://warp-drive.io/): The data layer (formerly Ember Data) that manages communication with the JSON:API backend via a request handler chain. ### Back-end Technologies The back-end is written in Java and uses the following technologies and frameworks: #### Pass Core * [Spring Boot](https://spring.io/projects/spring-boot): This is the back-bone of PASS, and the framework driving the development. Spring Boot simplifies the bootstrapping of PASS, by providing a methodology of convention over configuration for cleaner code and easier deployment. * [Elide](https://elide.io/): Exposes a JSON-API web service, which enables a versatile and standardized API for communication to the core logic of PASS. * [JPA](https://spring.io/projects/spring-data-jpa): The data access layer that communicates with the PostgresSQL database. Having Spring automatically wire up the interface to the database reduces boilerplate code and produces clean data access code. * [OCFL](https://github.com/OCFL/ocfl-java): An open implementation in Java, OCFL is used within PASS' file service to reliably store documents. It has support for both filesystem storage and AWS S3 integration. * [PostgreSQL](https://www.postgresql.org/): Serves as the database that stores all the information associated with grants, journals, publications, submissions, and policies. #### Data Loaders & Deposit Services In addition to using Spring Boot the Data Loaders and Deposit Services use the following technologies: * [SWORD](https://sword.cottagelabs.com/): This protocol is specifically utilized for the automated deposit of digital content into repositories. The SWORD protocol within PASS enables the standardized submission of scholarly works. In effect, providing compatibility and ease of integration with a variety of other repository platforms. * [Amazon SQS](https://aws.amazon.com/sqs/): Queues the deposit requests, allowing for asynchronous processing and ensuring that submissions are handled reliably. This decouples the submission process from the deposit execution, enhancing system resilience and scalability. ### Deployment and Hosting PASS is designed to be flexible and run on a variety of platforms; however at JHU we host PASS on Amazon Web Services. You can find more details about our deployment on our [Deployment Architecture](/welcome-guide/deployment-architecture) page. ### Development Tools and Practices * [Vite](https://vite.dev/) / [Ember CLI](https://cli.emberjs.com/release/): Vite is used as the build tool and dev server for pass-ui. Ember CLI provides scaffolding and test runner support. * [Docker](https://www.docker.com/): Utilized for running and testing PASS locally. * [TestCafe](https://testcafe.io/): The main framework of our PASS acceptance tests. It is a free and open-source solution for running end-to-end tests. * [Test Driven Development](https://en.wikipedia.org/wiki/Test-driven_development) & [JUnit](https://junit.org/): The Eclipse PASS team follows test driven development (TDD) to ensure high code quality and build confidence. Additionally, we use JUnit for unit tests and [Spring Boot Test](https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#features.testing)/[Testcontainers](https://testcontainers.com/) for integration tests. * [GitHub](https://github.com/): GitHub is where the source code for PASS is hosted. GitHub is also used to trigger deployments using a feature called GitHub Actions. # PASS Architecture PASS is designed to run on a variety of different platforms and architectures capable of running Spring Boot applications. This article describes one deployment strategy by highlighting the own Johns Hopkins University's application and deployment architecture. To achieve this deployment, several techniques are utilized including Continuous Integration and Continuous Deployment as known as CI/CD. ## PASS Application Architecture The PASS system's Amazon Web Services (AWS) application architecture employs a variety of AWS services to ensure high availability, security, and scalability. Two main components of the application architecture are the Elastic Cloud Compute (EC2) instances, which host the main application and the Elastic Container Service (ECS) containers, which are composed of the supporting Deposit services, Notification Services and Data Loaders. The diagram below provides a visual representation of JHU's deployment architecture:

JHU's Application Architecture

The main PASS application utilizes Elastic Compute Cloud (EC2) instances for Pass-core API and the Pass-UI. An Application Load Balancer (ALB) sits between these instances and Web Application Firewall (WAF), which gives security and stability to the PASS application. The ALB distributes incoming requests to appropriate target groups, which in turn route traffic to the correct EC2 instances running Pass-core and Pass-UI. The configuration of the Docker images running in EC2 comes from an S3 bucket which stores the configuration files. Persistence of the PASS application data is achieved by AWS Relational Database Service, and in particular using a PostgreSQL instance. All Docker images for the EC2 instances are stored in the Elastic Container Registry (ECR). The PASS application submits messages to the Simple Queue Service (SQS) when a submission is made. The Deposit Services listens for this and consumes those messages. This configuration, with the EC2 instances support a scalable and flexible environment when new application features are implemented and when user demand grows. Supporting the application infrastructure, AWS ECS Fargate hosts containerized services, namely the Deposit and Notification Services, which handle background processing and user interaction through scheduled tasks and real-time event responses. The AWS Batch service running on ECS, with dedicated computational environments for batch jobs, allows for efficient management of tasks such as data transfer from institutional repositories and JHU data sources. As mentioned earlier, the Deposit Services listens to the SQS and consumes the messages in the event of a submission and makes the appropriate deposit to the relevant institutional repository. This comprehensive setup ensures that the PASS system remains responsive and capable of scaling according to demand, while also adhering to organizational policies and data governance standards. ## PASS Deployment Architecture The deployment workflow starts when developers contribute code to the Eclipse PASS Git repository. Changes in this repository trigger GitHub Actions workflows, which are part of an automated CI/CD pipeline facilitating the deployment of the updated code. In deployment SQS is utilized for initiating the deployment from a GitHub workflow that publishes a SNS topic to the queue. The PASS deployment files contain configurations and environment variables that assist in the deployment of the PASS application and supporting services. Liquibase is utilized to manage database schema changes, which interacts with an AWS RDS instance running PostgreSQL.

JHU's Deployment Architecture

For other organizations looking to adopt a similar AWS application and deployment model, it's important to recognize that while the core architecture offers a template, it should be adapted to meet an organization's own requirements and needs. Each organization will need to evaluate its own application demands, data sensitivity, and user base to tailor the cloud resources, network configurations, and security policies accordingly. Moreover, integrating other types of cloud infrastructure or even on-premise solutions might be necessary to address specific technological preferences or regulatory requirements. Hybrid cloud environments or multi-cloud strategies could be employed to leverage the strengths of various cloud providers, enhance resilience, and avoid vendor lock-in. PASS is designed to be flexible and can adapt to a variety of architectures; whether a cloud infrastructure, hybrid or on-premise. # Latest Release We publish the latest release of PASS Core, Pass Support (Data Loaders, Deposit Services, Notification Services), PASS UI, and other supporting repositories to [GitHub](https://github.com/eclipse-pass). The artifacts from the Java based components are also published to a [Sonatype Maven Central Repository](https://central.sonatype.com/artifact/org.eclipse.pass/eclipse-pass-parent). Release announcements are made using a [Google Groups](https://groups.google.com/g/pass-general) mailing list, these announcements provide a brief summary highlighting the features, updates, and bug fixes that went into the release. ### GitHub * [PASS Parent (aka main)](https://github.com/eclipse-pass/main/releases) * [PASS Core](https://github.com/eclipse-pass/pass-core/releases) * [PASS Support](https://github.com/eclipse-pass/pass-support/releases) * [PASS UI](https://github.com/eclipse-pass/pass-ui/releases) * [PASS Docker](https://github.com/eclipse-pass/pass-docker/releases) * [Pass Acceptance Testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases) * [Pass Documentation](https://github.com/eclipse-pass/pass-documentation/releases) ### Sonatype During our release the Java based component artifacts for PASS are published to Sonatype Maven Central Repository: * [PASS Parent](https://central.sonatype.com/artifact/org.eclipse.pass/eclipse-pass-parent) * [PASS Core](https://central.sonatype.com/artifact/org.eclipse.pass/pass-core) * [PASS Support](https://central.sonatype.com/artifact/org.eclipse.pass/pass-support) ### SBOM As of release `1.13.0`, a [CycloneDX Software Bill Of Materials (SBOM)](https://cyclonedx.org/specification/overview/) is created and published for the following artifacts: * [pass-core-main](https://repo1.maven.org/maven2/org/eclipse/pass/pass-core-main/) * [deposit-core](https://repo1.maven.org/maven2/org/eclipse/pass/deposit/deposit-core/) * [pass-notification-service](https://repo1.maven.org/maven2/org/eclipse/pass/pass-notification-service/) * [pass-grant-loader](https://repo1.maven.org/maven2/org/eclipse/pass/pass-grant-loader/) * [pass-journal-loader-nih](https://repo1.maven.org/maven2/org/eclipse/pass/pass-journal-loader-nih/) * [nihms-data-harvest](https://repo1.maven.org/maven2/org/eclipse/pass/nihms-data-harvest/) * [nihms-data-transform-load](https://repo1.maven.org/maven2/org/eclipse/pass/nihms-data-transform-load/) * [pass-data-client](https://repo1.maven.org/maven2/org/eclipse/pass/pass-data-client/) * [pass-ui (Located in the / directory of the docker image)](https://github.com/eclipse-pass/pass-ui/pkgs/container/pass-ui) # Setup and Run PASS Locally ## Introduction The purpose of this article is to guide you through the process of getting PASS up and running locally using [Docker](https://www.docker.com/). If you're new to Docker, take a look at the [Get Started with Docker Guide](https://www.docker.com/get-started/). There are many other ways to deploy and run PASS, but this article focuses on getting PASS up and running on a local machine for a preview of the application. If looking to deploy PASS into a production environment, please take a look at the [PASS Infrastructure](/infrastructure-documenation) and [Developer Documentation.](/developer-documentation) ### Requirements * Docker * Docker Engine version 20.10.21 or higher * A minimum of 16GB of system memory is required, as Docker uses 8.5GB of virtual memory when starting up all services. ## Run PASS Locally ### Simple Setup Running PASS locally requires a few simple steps and no configuration, as defaults and demo data are already provided. If you do want to explore the configuration of PASS, the [configuration section](#configuration) of this page details how to edit the environment file. The first step is to clone or [download ](https://github.com/eclipse-pass/pass-docker/archive/refs/heads/main.zip)the code from the PASS Docker repository: {% code title="Cloning PASS Docker" %} ``` git clone https://github.com/eclipse-pass/pass-docker.git ``` {% endcode %} Once you've either cloned PASS docker or extracted it from the ZIP file, then you will want to use a command line tool to navigate to the root directory of PASS docker. 
cd C:\Users\username\IdeaProjects\pass-docker
Now that you're at the root directory of PASS Docker run the following command: {% code title="Run Docker Compose" overflow="wrap" %} ``` docker compose -f docker-compose.yml -f eclipse-pass.local.yml up -d --no-build --quiet-pull --pull always ``` {% endcode %} After running `docker compose` you should see the following images being pulled and started running in a container. {% code title="Docker Compose Output" %} ``` [+] Running 12/12 - Network pass-docker_front Created - Network pass-docker_back Created - Volume "pass-docker_db" Created - Container ldap Started - Container pass-ui Started - Container auth Started - Container proxy Started - Container localstack Started - Container pass-docker-postgres-1 Started - Container idp Started - Container pass-core Healthy - Container loader Started ``` {% endcode %} After the container is running, PASS is now running locally on your machine! All you need to do now is navigate to using a web browser, such as Firefox, Google Chrome, or Safari. From there you will see a login screen and can login using these test accounts:
User namePasswordDescription
nih-usermooUser that has NIH grants, and submissions on their behalf waiting for approval.
staff1mooUser that has one NIH Grant, and no submissions.
staff2mooUser that doesn't have any grants or submissions.
If you need to restart docker and re-run the containers, use the following command: {% code title="Docker Compose Down" %} ``` docker compose -p pass-docker down -v ``` {% endcode %} This will shutdown docker and remove all data associated with the application. This is important because when running `docker compose up` it loads fake data into the database. If you recreate the containers without previously destroying the data, duplicates will be inserted into the database. ### Advanced Setup The simple setup contains the workflow of creating a submission and simulating the deposit, but it doesn't actually go anywhere. If you want to test the full integration workflow with deposit services and submitting to DSpace or to NIHMS you can run the following command: {% code title="PASS with Deposit Services" overflow="wrap" %} ``` docker compose -p pass-docker -f docker-compose.yml -f eclipse-pass.local.yml -f docker-compose-deposit.yml -f docker-compose-dspace.yml up -d --no-build --quiet-pull --pull always ``` {% endcode %} Add an administrator and sample data into DSpace: {% code title="Add Administrator" overflow="wrap" %} ``` docker compose -p pass-docker -f dspace-cli.yml run --rm dspace-cli create-administrator -e test@test.edu -f admin -l user -p admin -c en ``` {% endcode %} {% code title="Add Sample Data" overflow="wrap" %} ``` docker compose -p pass-docker -f dspace-cli.yml -f dspace-cli.ingest.yml run --rm dspace-cli ``` {% endcode %} With Deposit Services and DSpace running in the container, when you run through the submission process and submit a manuscript to JScholarship it will make a deposit into DSpace. To view a deposit in the locally running DSpace instance navigate to in your web browser. Login using username:`test@test.edu` and password: `admin`. If submitting a deposit to [NIHMS](https://www.nihms.nih.gov), you can view the simulated deposit locally in the `pmc-sftp-server` image that is part of `pass-docker`. The first step is getting the name of the `pmc-sftp-server` container by running the `docker ps` command: ``` docker ps ``` It will output the following: {% code title="Docker ps Output" %} ``` CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c32ca1ae2905 dspace/dspace-solr:dspace-7.6 "/bin/bash -c 'init-…" 41 minutes ago Up 40 minutes 0.0.0.0:8983->8983/tcp dspacesolr fb2d2c0c46c7 ghcr.io/eclipse-pass/deposit-services-core:1.6.0-SNAPSHOT "./entrypoint.sh" 41 minutes ago Up 40 minutes pass-deposit-services 8394fa5b55da ghcr.io/eclipse-pass/idp:1.6.0-SNAPSHOT "./entrypoint.sh" 41 minutes ago Up 41 minutes 4443/tcp, 8443/tcp idp 65e71a8770d4 dspace/dspace:dspace-7.6-test "/bin/bash -c 'while…" 41 minutes ago Up 40 minutes 8000/tcp, 8009/tcp, 0.0.0.0:8080->8080/tcp dspace df73b4336089 ghcr.io/eclipse-pass/pass-core-main:1.6.0-SNAPSHOT "./entrypoint.sh" 41 minutes ago Up 40 minutes (healthy) pass-core 015446cc79c1 ghcr.io/eclipse-pass/pass-auth:1.6.0-SNAPSHOT "docker-entrypoint.s…" 41 minutes ago Up 41 minutes 80/tcp, 443/tcp auth 0b7b9956da40 localstack/localstack:3.2.0 "docker-entrypoint.sh" 41 minutes ago Up 40 minutes (healthy) 127.0.0.1:4510-4559->4510-4559/tcp, 127.0.0.1:4566->4566/tcp, 5678/tcp localstack 2ab832b2ec61 postgres:14-alpine "docker-entrypoint.s…" 41 minutes ago Up 41 minutes 5432/tcp pass-docker-postgres-1 1b9fb04ddb02 dspace/dspace-postgres-pgcrypto:dspace-7.6 "docker-entrypoint.s…" 41 minutes ago Up 41 minutes 0.0.0.0:5432->5432/tcp dspacedb 6e8230eb7814 ghcr.io/eclipse-pass/pass-ui:1.6.0-SNAPSHOT "/bin/entrypoint.sh" 41 minutes ago Up 41 minutes 80/tcp pass-ui 84b402f8255c ghcr.io/eclipse-pass/demo-ldap:1.6.0-SNAPSHOT "./entrypoint.sh" 41 minutes ago Up 41 minutes 389/tcp ldap 1510d03d1f08 dspace/dspace-angular:dspace-7.6 "docker-entrypoint.s…" 41 minutes ago Up 41 minutes 0.0.0.0:4000->4000/tcp, 0.0.0.0:9876->9876/tcp dspace-angular 24e1ee4ac934 atmoz/sftp "/entrypoint pmcsftp…" 41 minutes ago Up 41 minutes 0.0.0.0:2222->22/tcp pmc-sftp-server 4253aa6924a0 ghcr.io/eclipse-pass/proxy:1.6.0-SNAPSHOT "entrypoint.sh" 41 minutes ago Up 40 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp proxy ``` {% endcode %} Get the container ID that is associated with `atmoz/sftp.`Using the container ID enter the SFTP container. {% code title="Get the Container ID" %} ``` docker ps -aqf name=pmc-sftp-server ``` {% endcode %} Copy the result from the "Get Container ID" command. Then run the "Enter the SFTP Container" command by replacing the "PasteContainerIDHere" with the previously copied container ID. {% code title="Enter the SFTP Container" %} ``` docker exec -it PasteContainerIDHere /bin/bash ``` {% endcode %} Now that you're in the SFTP server, navigate to the deposit. **Note:** the last directory that is a date will be the date when the deposit is made. ``` cd /home/pmcsftpuser/upload/2024-04-02 ``` And you will see all the deposits that you've made to NIHMS: ``` root@24e1ee4ac934:/home/pmcsftpuser/upload/2024-04-02# ls nihms-native-2022-05_2024-04-02_19-04-00_68.zip nihms-native-2022-05_2024-04-02_19-04-48_68.zip nihms-native-2022-05_2024-04-02_19-04-57_68.zip nihms-native-2022-05_2024-04-02_19-04-58_68.zip ``` Congratulations! You've simulated a manuscript deposit to the institutional repository (JScholarship) and NIHMS! ## Configuration All the defaults in the docker env configuration files will run without any modification, but if you need to change a port, URLs, or other configurations you can do so using the `.eclipse-pass.local_env` file. It's recommend while testing to keep these values the defaults, but if you need to change a port for a specific reason you can do so by modifiying the `.eclipse-pass.local_env`environmental variable file. ## Troubleshooting 1. **Docker Compose Fails to Start the Services** **Problem**: Users might encounter errors when running the `docker compose up` command due to various reasons such as network issues, Docker daemon not running, or insufficient permissions. **Solution**: Ensure Docker is running on your machine. Check your internet connection and firewall settings. If Docker is running and no networking connections issues are present, then trying updating or installing the [latest version of Docker](https://www.docker.com/get-started/). *** 2. **Insufficient System Memory Error** **Problem:** The application might fail or perform poorly if the system does not meet the minimum memory requirement. **Solution:** Close unnecessary applications to free up memory. Consider increasing your system's memory if persistent issues occur. Ensure you have at least 16GB of system memory available as recommended. If you're running Docker on a Windows machine, ensure that WSL2 and Docker Compose V2 are enabled in the settings. *** 3. **Unable to Access PASS on the web browser** **Problem:** After running the Docker compose command, the PASS application does not load or displays an error in the web browser. **Solution:** Verify that the containers are running by running the `docker ps` command. When running without deposit services you should see the following containers running(**Note**: image version may differ as it will be updated in the future e.g. 1.6.0-SNAPSHOT): ``` CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 813972bad937 ghcr.io/eclipse-pass/idp:1.6.0-SNAPSHOT "./entrypoint.sh" About a minute ago Up 56 seconds 4443/tcp, 8443/tcp idp 2aafc2de282e ghcr.io/eclipse-pass/pass-core-main:1.6.0-SNAPSHOT "./entrypoint.sh" About a minute ago Up 43 seconds (healthy) pass-core 960fbbed7458 ghcr.io/eclipse-pass/pass-auth:1.6.0-SNAPSHOT "docker-entrypoint.s…" About a minute ago Up 57 seconds 80/tcp, 443/tcp auth db263a61122d localstack/localstack:3.2.0 "docker-entrypoint.sh" About a minute ago Up 44 seconds (healthy) 127.0.0.1:4510-4559->4510-4559/tcp, 127.0.0.1:4566->4566/tcp, 5678/tcp localstack 663871b8a92b postgres:14-alpine "docker-entrypoint.s…" About a minute ago Up 58 seconds 5432/tcp pass-docker-postgres-1 b12319e6ccbb ghcr.io/eclipse-pass/pass-ui:1.6.0-SNAPSHOT "/bin/entrypoint.sh" About a minute ago Up 58 seconds 80/tcp pass-ui a51102277cff ghcr.io/eclipse-pass/demo-ldap:1.6.0-SNAPSHOT "./entrypoint.sh" About a minute ago Up 58 seconds 389/tcp ldap 576ff2aae621 ghcr.io/eclipse-pass/proxy:1.6.0-SNAPSHOT "entrypoint.sh" About a minute ago Up 54 seconds 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp proxy ``` When running with Deposit-Services and DSpace, as specified in the [advanced setup](#advanced-setup), you will see the following containers running: ``` CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 8359f7400413 ghcr.io/eclipse-pass/idp:1.6.0-SNAPSHOT "./entrypoint.sh" 41 seconds ago Up 33 seconds 4443/tcp, 8443/tcp idp 0d0abd03c480 dspace/dspace-solr:dspace-7.6 "/bin/bash -c 'init-…" 41 seconds ago Up 26 seconds 0.0.0.0:8983->8983/tcp dspacesolr b14227cec259 ghcr.io/eclipse-pass/deposit-services-core:1.6.0-SNAPSHOT "./entrypoint.sh" 41 seconds ago Up 9 seconds pass-deposit-services 238a6183e1ab ghcr.io/eclipse-pass/pass-core-main:1.6.0-SNAPSHOT "./entrypoint.sh" 41 seconds ago Up 12 seconds (health: starting) pass-core a7a24772c12d dspace/dspace:dspace-7.6-test "/bin/bash -c 'while…" 41 seconds ago Up 29 seconds 8000/tcp, 8009/tcp, 0.0.0.0:8080->8080/tcp dspace bc3ea5a2ec20 ghcr.io/eclipse-pass/pass-auth:1.6.0-SNAPSHOT "docker-entrypoint.s…" 43 seconds ago Up 35 seconds 80/tcp, 443/tcp auth a7403c2b34a6 localstack/localstack:3.2.0 "docker-entrypoint.sh" 43 seconds ago Up 15 seconds (healthy) 127.0.0.1:4510-4559->4510-4559/tcp, 127.0.0.1:4566->4566/tcp, 5678/tcp localstack 135dc909b2c1 postgres:14-alpine "docker-entrypoint.s…" 43 seconds ago Up 35 seconds 5432/tcp pass-docker-postgres-1 f8c24a38d438 ghcr.io/eclipse-pass/proxy:1.6.0-SNAPSHOT "entrypoint.sh" 43 seconds ago Up 29 seconds 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp proxy d9bb301597f4 ghcr.io/eclipse-pass/demo-ldap:1.6.0-SNAPSHOT "./entrypoint.sh" 43 seconds ago Up 37 seconds 389/tcp ldap 340435bf059f ghcr.io/eclipse-pass/pass-ui:1.6.0-SNAPSHOT "/bin/entrypoint.sh" 43 seconds ago Up 35 seconds 80/tcp pass-ui e6137acc145a dspace/dspace-angular:dspace-7.6 "docker-entrypoint.s…" 43 seconds ago Up 32 seconds 0.0.0.0:4000->4000/tcp, 0.0.0.0:9876->9876/tcp dspace-angular 778c5cec16d7 atmoz/sftp "/entrypoint pmcsftp…" 43 seconds ago Up 33 seconds 0.0.0.0:2222->22/tcp pmc-sftp-server 66daca298f22 dspace/dspace-postgres-pgcrypto:dspace-7.6 "docker-entrypoint.s…" 43 seconds ago Up 34 seconds 0.0.0.0:5432->5432/tcp dspacedb ``` If some of the containers are not running try `docker compose -p pass-docker down -v` and restart the containers by running the docker compose command mentioned in the [simple ](#simple-setup)or[ advanced setup](#advanced-setup) sections. # Collaboration with Other Institutions PASS is a community oriented Open Source Project and we're always looking to collaborate with other institutions and individuals that believe in [Open Access](https://en.wikipedia.org/wiki/Open_access). The PASS team has engaged in extensive collaborative efforts with various academic institutions, hosting providers, and government agencies to advance the development and integration of the PASS system. These collaborative initiatives were structured around two central themes: community engagement and communications. ### Community Engagement In the realm of community development, the PASS team, alongside members from the Eclipse Foundation, organized bi-weekly meetings during the 2023 Fall semester. These meetings fostered a collaborative environment with academic partners and hosting providers interested in the PASS system. Participants included organizations such as [CalTech](https://www.caltech.edu/), the [University of Virginia](https://www.virginia.edu/), the [University of Oregon](https://www.uoregon.edu/), the [University of Louisville](https://louisville.edu/), and the [National Center for Atmospheric Research (NCAR)](https://ncar.ucar.edu/), along with potential hosting providers [Lyrasis ](https://www.lyrasis.org/Pages/Main.aspx)and [TIND](https://www.tind.io/ir). The primary focus of these gatherings was to understand the challenges PASS could address, define software improvements for a PASS pilot, and better understand the deployment environments. This collaborative effort culminated in identifying pilot opportunities for 2024 and pinpointing institutional uses cases and integration requirements. ### Communications The PASS project team also undertook a campaign to spread awareness and foster discussions about PASS with a wide array of groups and institutions. Notable engagements included discussions on PASS integration with the National Institutes of Health (NIH) and the National Science Foundation (NSF), presentations to the HELIOS Infrastructure Working Group and the Big Ten Academic Alliance, and participation in various spotlight series and conferences. These communications extended throughout the year, touching base with organizations such as SPARC, the US Repository Network, Figshare, and the Invest in Open Infrastructure initiative, culminating in a presentation at the [Fall 2023 Coalition for Networked Information (CNI)](/welcome-guide/pass-demonstrations-conferences#cni-presentation-december-2023) and discussions with the NASA/CERN open science working group in early 2024. ### Looking Forward As we move forward into 2024, the PASS team is focused on facilitating the technical trials within the PASS community. These tasks are critical for enhancing the PASS system's functionality and ensuring its integration into broader academic and research infrastructures. The PASS team aims to enhance the PASS software by importing grant data and associated user data, developing a simple administrative user interface, adding additional user authentication options, and establishing a community testing environment. In addition there are two focused collaborations: 1. **Institutional Repository Deposit Integrations:** In collaboration with CalTech, integrate PASS with the repository [InvenioRDM](https://inveniordm.docs.cern.ch/). 2. **Integrate with a Funder Repository:** The team plans to engage with NSF PAR and DOE OSTI, with support from other interested groups such as Stanford, NCAR, and Dryad. This integration is pivotal for broadening the PASS ecosystem and enhancing its utility for various stakeholders. These strategic directions underscore our commitment to evolving PASS into a more robust, versatile platform that meets the needs of the academic and research communities. As we embark on these tasks, our focus remains on collaboration, innovation, and the continued pursuit of excellence in supporting open science and research endeavors. If you or your institution is interested in collaborating with the PASS Team, please send an email to our community [Google Group](mailto:pass-general@googlegroups.com). # Contributing to PASS ### Getting Involved Whether you are interested in simply improving the existing code base, or maybe even thinking about forking the project for use at your institution, it's a good idea to have a look at a running test instance and exercise the workflows. The easiest way to do this is to simply visit our [demo instance](https://demo.eclipse-pass.org/). ### Reporting a Bug If you've run across a bug in the Eclipse PASS application, letting us know about it is an important and an easy way to contribute. First, check to see if there's an existing issue for what you're observing by performing a search in the [issues list](https://github.com/eclipse-pass/main/issues). If you've discovered a new bug, create a new issue to tell us about it. You will need to create a [GitHub ](https://github.com/)account to do this, if you don't have one already. When creating an issue, please provide a concise description of the potential problem, the steps needed to reproduce it, and a description of how the behavior differs from what you need or expect. Also, provide details about your environment like OS, Java version, and Maven version, etc. ### Contributing Code or Documentation There is always a need for fixing bugs and we are continually looking to add new features. Contributions furthering these efforts are always welcome. It may be that you simply notice something that doesn't seem to work quite right or the list of issues on our [SonarQube Cloud](https://sonarcloud.io/organizations/eclipse-pass). You can check the GitHub [issues ](https://github.com/eclipse-pass/main/issues)to see if an issue has already been opened. If an issue was already opened, you may also comment on the issue if you have new information which might be helpful. If there is no existing issue, you might consider creating one ([see above](#reporting-a-bug)). The same process holds for exploring the addition of a new feature or updating documentation. To contribute to the code base, you will need an account on GitHub. Further, since PASS is an Eclipse Foundation project, contributors will need to [create an Eclipse account](https://accounts.eclipse.org/) and sign a [contributor agreement](https://www.eclipse.org/legal/ECA.php). To keep things simple, the email address you use to sign up for the Eclipse account should be the same as the one you use for your GitHub account. To make changes to code or documentation, you'll start by forking the GitHub repository you'd like to change into your own GitHub account. Once you have your own copy of the repository, create a Git branch for your work. Please name the branch based on the ID of the ticket associated with the issue, for example: `903-fix-bug-in-code`. You'll then be able to make changes, perform Git commits, and push those changes to your copy of the repository. Once your work is done, you'll need to create a Pull Request to let us know about your changes. [You'll find a more complete overview of this process here](https://opensource.com/article/19/7/create-pull-request-github). ### More Info More documentation about the PASS project can be found in the [Developer Documentation](/developer-documentation) and the [PASS Infrastructure](/infrastructure-documenation) sections of this documentation repository. General information about contributing to Eclipse projects can be found in the [Eclipse handbook](https://www.eclipse.org/projects/handbook/#contributing-contributors). # Community The PASS Community Documentation provides guidelines and resources for contributing to the Eclipse PASS project,\ covering key areas such as [developer guidelines](/community/developer-guidelines), the [project roadmap](/community/pass-roadmap), and\ ways to get involved. It outlines best practices for reporting bugs, contributing code, and testing to ensure a\ high-quality and collaborative development process. The documentation also includes the PASS roadmap, which details\ planned initiatives around user engagement, community building, system administration, and application enhancements. ## Getting Involved PASS is an open-source community project, and there are many ways to contribute—whether by providing feedback, reporting\ bugs, or writing code. Community members are encouraged to: * Review the existing code base and documentation. * Test the [PASS Docker](/developer-documentation/pass-docker) local test instance to understand workflows and features. * Join discussions on [Slack](https://eclipse-pass.slack.com/archives/C035MNLRD44) to ask questions or offer suggestions. * Follow the [pull request workflow](/community/developer-guidelines#pull-request-workflow) to submit code contributions. If you’re new to the project, a great starting point is to try the [PASS Docker](/developer-documentation/pass-docker)\ local test instance to explore the application and understand its core functionality. For more detailed developer information, visit the [PASS Developer Documentation](/developer-documentation) and read\ over the [developer guidelines](/community/developer-guidelines). To review the PASS roadmap and stay updated on future\ developments, see our [PASS Roadmap](/community/pass-roadmap). # Developer Guidelines The PASS Developer Guidelines provide detailed instructions for contributing to the PASS project, covering areas such as communication channels, testing procedures, pull request workflows, and documentation standards. It outlines expectations for contributors, the process for reporting issues, and best practices for maintaining code quality and project integrity. ## Getting Involved * Welcome to the PASS community! There are many ways to participate: trying out the PASS software, letting us know about bugs, suggesting documentation updates, or contributing code. After you’ve read this guide, if you have questions, please send us a message on our [Google Group](https://groups.google.com/g/pass-general), and we will be in touch shortly! * Looking for a place to start contributing? Our [SonarQube Cloud](https://sonarcloud.io/organizations/eclipse-pass) has a list of bugs/issues. * We primarily use Slack to communicate about PASS development. To be invited to our Slack workspace, please send us a message on our [Google Group](https://groups.google.com/g/pass-general). * Contributing to the project begins as a [contributor](https://www.eclipse.org/projects/handbook/#contributing-contributors) and may lead to being a [committer](https://www.eclipse.org/projects/handbook/#roles-cm). Whether you are a `contributor` or `committer`, you will need to sign up for an [Eclipse account](https://accounts.eclipse.org/user/login). * A `contributor` can add to and improve PASS by creating issues and submitting pull requests. You’ll find more information about both of these tasks below. * A `committer` is an individual who once was a `contributor`, but made significant contributions and was elected by the core team to become a `committer`. They can work directly in the repositories, create and close issues, and merge pull requests. ## Change Request/Bug Report Would you like to suggest a change to PASS or report a bug? This is done by submitting a GitHub issue. * When creating an issue to report a bug or suggest a new feature, use the [eclipse-pass/main repository](https://github.com/eclipse-pass/main/issues). * If available for your particular issue use one of the available [issue templates in the main repository](https://github.com/eclipse-pass/main/issues/new/choose). * If a suitable template doesn’t exist, use the default `Standard Issue` template. * Add a label if possible, if the label doesn’t exist, or you’re unsure of which one to use, send a message to the team on the Slack `#pass-dev` channel. * If possible, suggest a priority. If you’re unsure, leave it blank, and the team will determine the appropriate priority. ## Testing In general, we recommend the following procedures for testing: * If you're planning to submit code through a Pull Request (PR), please run tests locally first. For Java code, this can be done using `mvn verify` or `mvn clean install`. To test the complete project, [run pass docker](/welcome-guide/setup-run-pass) to test. * If you're planning to submit code which includes new tests, Martin Fowler’s [The Practical Testing Pyramid](https://martinfowler.com/articles/practical-test-pyramid.html) is a great resource for understanding how to structure tests. Additionally, we use this [definition for ITs](https://www.geeksforgeeks.org/software-engineering-integration-testing/) along with [Martin Fowler's definition](https://martinfowler.com/bliki/IntegrationTest.html). PASS has three different types of tests that are run against the application, and they are defined as: * **Unit Tests**: Unit tests focus on a single unit of code. They test very specific conditions, inputs, and expected outputs, validating that the unit behaves as intended. They are narrow in scope, and all other collaborators (e.g. other classes that are called by your class under test) are substituted with mocks or stubs. Unit tests alone do not guarantee the application as a whole will work as intended. * **Integration Tests**: They test the integration of your application with other parts that are not part of your application e.g. databases, external REST APIs. They are not as narrow as Unit Tests, but still test one integration point at a time. In addition, ITs focus on verifying the interactions and data exchange between different components or modules of a software application. * **Acceptance Tests**: They are a final validation step, ensuring PASS fits the workflow requirements for users. The [PASS Acceptance Tests](https://github.com/eclipse-pass/pass-acceptance-testing) runs through workflows using [Test Cafe](https://testcafe.io/) against an instance of PASS. All these tests must pass in order for PASS to be considered production ready. ### Back-end * **Unit Tests** * If introducing new functionality, please ensure new code is covered by at least one unit test which includes both success and failure states. * A few examples of this are [here](https://github.com/eclipse-pass/pass-support/blob/79ad19ed4d2592c342e7cdfdf652a8f7aef3eaa2/pass-deposit-services/deposit-core/src/test/java/org/eclipse/pass/deposit/service/DepositProcessorIT.java#L54) and [here](https://github.com/eclipse-pass/pass-core/blob/e9e853ac7eea05f595fdcd5342ddea99c0798e38/pass-core-main/src/test/java/org/eclipse/pass/object/ElidePassClientTest.java#L84). * If performing a bug fix, include a test to ensure that the bug was fixed. * In general, unit tests should run quickly. * **Integration Tests** * We recommend running integration tests in a test environment that mimics the production environment as closely as possible. * When adding or updating integration tests, please avoid making network requests to 3rd parties. * If needed, use test containers, wiremock, or mockbean. * Integration tests should be as fast as possible. * **Acceptance Tests** * Acceptance tests are used in PASS to verify correct functionality based on user requirements, so these should work correctly from a user’s perspective. * Updated whenever there are changes to user requirements, significant changes are made to the application, or when they break. * Automated so they can be run frequently and consistently. ### UI * When testing the UI it is helpful to run [Ember locally for faster iteration](https://github.com/eclipse-pass/main/blob/main/docs/dev/running-pass-ui-on-your-host-machine.md). * Include a unit test when you can, such as when functions don't interact with rendering. Otherwise, utilize component integration or ember application/acceptance tests where rendering is involved - this is what ember is best at. * [Pass-ui](https://github.com/eclipse-pass/pass-ui) is heavy on integration/application tests because it's rendering heavy and much of the business logic is in the back end. * If you write an encapsulated piece of UI like a component, that component should have at least 1 integration test. * Application level testing is done with mocked data using Mirage. This needs to be updated diligently, so it doesn't fall out of sync with the real back end. If you are updating the API in a way that changes the contract with pass-ui, please create an issue for updating the UI mocking to accommodate these changes. * At least one test should be added for bug fixes to prevent regression. * The [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing) acceptance tests are run frequently and can aid in making sure the UI application testing mocked responses are appropriate. * Ember provides a set of very helpful libraries to assist in testing UI components. It is best practice to use these helpers rather than inventing your own where possible: * [Ember test-helpers](https://github.com/emberjs/ember-test-helpers/blob/master/API.md) * [QUnit-dom](https://github.com/mainmatter/qunit-dom/blob/master/API.md) ## Commits * As a team, we do not enforce rigid rules about commit messages. However, we strive to write good commit messages by following these [guidelines by Chris Beams](https://cbea.ms/git-commit/). ## Documentation * We encourage you to read through the [PASS documentation style guide](https://docs.google.com/document/d/11aCooQCNhEq34yG9mGuFynY4xMDRJtPRuOOou9LaCgU/edit?usp=sharing) prior to submitting a pull request. * The PASS team uses [GitBook](https://www.gitbook.com/) for managing and creating documentation. There are two ways to create new documentation with this system, through the GitBook web interface and through our GitHub `pass-documentation` repository. * The process for creating, editing, and managing documentation will vary depending on which system you use: * GitHub: * Use a personal branch that is checked out from `development` and is rebased back into development. * Ensure the branch is up-to-date with `development` before creating a pull request. * Follow the same pull request guidelines mentioned in our Pull Request Workflow section. * GitBook: * Request to be added to the GitBook team. * Follow the change request process as outlined by the GitBook docs. * The same pull request guidelines mentioned in the Pull Request Workflow section apply here as well, such as who should be the reviewer. * **NOTE**: It is easy to merge directly from GitBook, ensure that the button at the top right is changed from `merge` to `request a review`. * Each repository in the PASS project should have a top level README. The following guidelines for these readme should include: * Overview of project purpose. * Links to appropriate sections in GitBook. * README in the main repo should include more details and a longer overview of the PASS project. ## Pull Request Workflow * If you are in the `contributor` role, you must first fork the repository you wish to make changes in and then submit the pull request to the upstream. If you’re not familiar with pull requests, see the [GitHub pull request documentation](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests). * Pull requests should be reviewed by at least one `committer` that is not the PR author before merging. * For pull requests created by `committers`, the pull request author is expected to perform the pull request merge after another committer has approved the pull request. * For pull requests created by contributors, the committer reviewing the pull request is responsible for merging the pull request after approval. * Ensure your branch is created from the latest version of `main`. * Branch names should include a ticket number and short description. * Example: `978-fix-nihms-loader-etl` * The description in your pull request should include the following: * Summary of major changes and what the pull request will accomplish. * Instructions identifying how to test the changes. * Ensure that every PR is linked to a relevant ticket. * Update or add new documentation to the `pass-documentation` repository. * This would be a separate pull request, see the Documentation section for this process. * A pull request should not be merged unless all automated checks pass. * The [SonarQube quality gates](/infrastructure-documenation/sonar-qube) are optional, but it is encouraged to address these code quality checks. * Merges should happen using the rebase strategy. * If there have been changes to the `main` code branch, you may want to rebase your branch on `main` for additional safety. * After a successful merge, delete the branch. ## Pull Request Review Process * When reviewing the code, here are some advised areas to consider: * Verify that the implemented logic aligns with the requirements and specifications. * Look for any potential bugs or logical errors. * Ensure code is adequately commented where necessary. * Verify that sensitive configurations are externalized and not hard-coded. * Ensure REST endpoints follow standard conventions (e.g., proper use of [HTTP methods](https://developer.mozilla.org/en-US/docs/Web/HTTP)). * Review the [SonarQube quality gates](/infrastructure-documenation/sonar-qube) and [JaCoCo code coverage](/infrastructure-documenation/sonar-qube/jacoco) reports. * Review any unit/integration tests and ensure that they provide proper coverage. * Ensure proper use of mocks and stubs to isolate components during testing. * Identify and suggest refactoring for any [code smells](https://linearb.io/blog/what-is-a-code-smell). * Look for areas that could benefit from improved [design patterns or structures](https://www.baeldung.com/design-patterns-series). * Ensure that dependencies are properly managed and up-to-date. * Check for any potential conflicts or unused dependencies. * Review commit messages. * Build the project and run tests locally. ## Closing Issues * If an issue is linked to a pull request it will auto-close, however for issues that are not linked to a pull request, the committer performing the merge should close the completed issues. * Write up the final outcome in the issue and include this as a comment when closing the ticket. * Link any collaboration or design documents in the issue before closing. * Ensure all related PRs are linked and closed. * We recommend that the changes are deployed and tested in a staging or preproduction environment prior to completion. ## Protecting Sensitive Information * Take precaution to ensure that you’re not committing any credentials/keys/secrets. * If any sensitive information is accidentally committed, immediately notify the team on the `#pass-dev` Slack channel. * The core team will triage the severity of the leak and take appropriate actions. This may include removing it from the GitHub and GitBook commit history and rotating the compromised credentials. * Update GitHub secret scanning to catch any sensitive information that bypassed the original scan. # PASS Roadmap ## Eclipse PASS Project Roadmap This roadmap defines the primary initiatives of the Eclipse PASS Project, organized by anticipated release. If you are interested in helping to define and/or contribute to this roadmap, please reach out to us on our [Google Group](https://groups.google.com/g/pass-general). We're always happy to welcome new contributors! ### Priorities and goals #### User Engagement * Engage with current and potential PASS users to understand their needs and how PASS can better address their challenges. #### Community Engagement * Engage with and build out the broader community of institutions that have interest in utilizing PASS. * Provide a baseline method for other institutions to import the necessary data into PASS. * Prepare PASS to be deployed, used, and implemented at other institutions. * Add at least one additional IR deposit target. #### System Administration * Provide improved tooling and support for administering the PASS system. * Improve deployment infrastructure. * System monitoring and alerting. * Design and mock up an administrative dashboard that visualizes submissions and provides key statistics of the PASS application. #### Application Improvements * Integrate with at least one additional funder repository. * Improve code workflow using Continuous Integration and Continuous Delivery / Deployment. * Ensure the NIH integration is solid. * Work on technical debt and ensure system components are kept up-to-date. * Security and testing. #### Documentation * Improve documentation to prepare for collaboration with external institutions. # Release Notes ### Release v2.5.1 #### Date: April 13, 2026 Release Manager: Russ Poetker, JHU This release includes updates to resolve Critical CVEs in third-party dependencies. Additionally, Ember was upgraded to the latest LTS. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/40?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/2.5.1) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/2.5.1) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/2.5.1) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/2.5.1) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/2.5.1) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/2.5.1) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/2.5.1) ### Release v2.5.0 #### Date: March 30, 2026 Release Manager: Russ Poetker, JHU This is a maintenance release for PASS. Frontend and Backend dependencies were upgraded. In the pass-ui project, Ember was upgraded to the latest LTS and converted to Typescript. The Journal Data Loader was updated to support the new PMC file format. A bug related to DOI lookup error handling was fixed. Additionally, GitHub Actions workflows were updated to use immutable versions for third-party actions. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/39?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/2.5.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/2.5.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/2.5.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/2.5.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/2.5.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/2.5.0) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/2.5.0) ### Release v2.4.0 #### Date: November 18, 2025 Release Manager: Russ Poetker, JHU This release includes a change to the Deposit Services DSpace integration to support DSpace 9.x. There is also a change in the UI messaging related to PMC submission terminology. Additionally, the children POMs were cleaned up to remove inherited attributes. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/38?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/2.4.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/2.4.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/2.4.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/2.4.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/2.4.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/2.4.0) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/2.4.0) ### Release v2.3.0 #### Date: September 23, 2025 Release Manager: Russ Poetker, JHU This release includes two areas of change. The first is a set of changes to improve messaging to users describing what happens after the completion of a submission. Now an appropriate message in the UI will describe the next steps depending on the submission's target repositories. There were also several updates made to the grant selection tables with regard to improving UX and fixing a few bugs. The second area of change is in the grant loader where a new integration with the Fibi Grant Management System has been added. The PASS release workflow has also migrated to Central Portal for publishing artifacts to Maven central. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/36?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/2.3.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/2.3.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/2.3.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/2.3.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/2.3.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/2.3.0) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/2.3.0) ### Release v2.2.0 #### Date: May 28, 2025 Release Manager: Jared Galanis, JHU This release includes a refactoring of the way metadata schemas are processed and handled in pass-ui. This was a broad refactor that improved and simplified metadata state management and processing in pass-ui, replaced an outdated library (alpaca.js) in pass-ui that previously handled metadata forms with a more modern JavaScript library (survey.js), and moved the metadata schema service from pass-core into pass-ui. This release also removed support for the pass demo site, updating documentation and shutting down AWS resources that served the demo site. This release additionally resolved intermittent failures in pass-acceptance-testing CI runs and improved the documentation around the release process itself. During this release cycle the team also investigated and supported the migration from Sonatype OSSRH (being deprecated at the end of June) to Central Portal for publishing artifacts to Maven central. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/35?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/2.2.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/2.2.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/2.2.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/2.2.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/2.2.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/2.2.0) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/2.2.0) ### Release v2.1.1 #### Date: May 6, 2025 Release Manager: Mark Patton, JHU This is a patch release to fix a bug viewing the submission details page and fix a bug that prevented submissions with an embargo to DSpace from working. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/37?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/2.1.1) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/2.1.1) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/2.1.1) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/2.1.1) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/2.1.1) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/2.1.1) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/2.1.1) ### Release v2.1.0 #### Date: April 30, 2025 Release Manager: Jared Galanis, JHU This release introduces support in pass-core for Spring Cloud AWS to connect to S3 for reading config files. It also addresses technical debt in pass-support, includes test cleanup, and removes the MD5 checksum option in deposit services. Additionally, it resolves a page reload bug and a submit confirmation bug in pass-ui, along with refactoring the logic for submission file state management. Finally, documentation for SonarQube and JaCoCo has been added in this release. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/34?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/2.1.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/2.1.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/2.1.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/2.1.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/2.1.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/2.1.0) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/2.1.0) ### Release v2.0.0 #### Date: March 27, 2025 Release Manager: Tim Sanders, JHU This is a major release of PASS, marking the transition from the 1.x series to 2.x. It includes important security updates, new tools for improving code quality, and removed SWORD deposits. A research spike was also conducted to explore removing AlpacaJS. We had our first pull request from an external contributor with the [Pass-Docker InvenioRDM update](https://github.com/eclipse-pass/pass-docker/pull/389)! Highlights * Removed SWORD Support: SWORD deposit support has been removed. * Security Fixes: Updated dependencies identified with CVEs, improving the security posture of the PASS user interface. Strengthened Content Security Policy (CSP) headers to mitigate risks such as cross-site scripting (XSS) and content injection attacks. * Code Coverage Setup: Configured SonarQube to ingest JaCoCo reports, providing real-time code coverage status checks for pull requests. * Remove Alpaca Analysis: Investigated the feasibility of removing AlpacaJS from the metadata step, exploring potential improvements and simplifications for dynamic form generation. Additional Changes * Bug Fix: Addressed an issue with the proxy search dialog displaying at the top of the page rather than in its intended dialog. * Cleanup Documentation: Archived out-dated documentation. * Pass-Docker InvenioRDM: Update InvenioRDM to v12 in local docker environment. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/32?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/2.0.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/2.0.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/2.0.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/2.0.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/2.0.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/2.0.0) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/2.0.0) ### Release v1.15.0 #### Date: February 27, 2025 Release Manager: Mark Patton, JHU This release adds support for transforming JATS XML abstracts imported from a DOI to HTML for better display in a repository. Dependencies have been updated to address security problems found by an analysis of SBOMs. Fixes were made handling cleanup of test deposits which use the DSpace API. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/30?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.15.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.15.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.15.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.15.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.15.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.15.0) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/1.15.0) ### Release v1.14.1 #### Date: February 12, 2025 Release Manager: Russ Poetker, JHU This release is a patch release to fix a bug in notification services that was not allowing emails to be sent. This release also contains the new DSpace API transport in deposit services which is the new way to perform deposits into DSpace. The DSpace API transport will eventually replace the SWORD transport for DSpace deposits. Additionally, pass-docker was changed so that test PKI files are generated as needed on startup for the test IDP and InvenioRDM containers. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/31?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.14.1) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.14.1) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.14.1) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.14.1) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.14.1) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.14.1) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/1.14.1) ### Release v1.14.0 #### Date: January 29, 2025 Release Manager: Russ Poetker, JHU This release adds SonarQube Cloud for pass-core, pass-support, and pass-ui for Static Code Analysis to improve code quality and security (badges have been added to each project showing SonarQube Quality Gate status). The pass-core and pass-support projects now produce JaCoCo code coverage metrics. In a near-term future release, code coverage reports will be generated on SonarQube Cloud. The failed deposit retry rule has changed so that a failed deposit is only automatically retried if the target repository was unreachable. The release workflow has been updated to release the pass-documentation project. Various code cleanups and dependency updates have been made to the project. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/29?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.14.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.14.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.14.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.14.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.14.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.14.0) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/1.14.0) ### Release v1.13.0 #### Date: December 9, 2024 Release Manager: Mark Patton, JHU This release added SBOM creation to our release process. GitHub repository documentation now points to the new documentation site. Various code cleanups and dependency updates have been made to the Java backend. In addition work has started on support for direct deposit with the DSpace REST API. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/28?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.13.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.13.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.13.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.13.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.13.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.13.0) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/1.13.0) ### Release v1.12.0 #### Date: October 31, 2024 Release Manager: Tim Sanders, JHU With this version, we’ve implemented improvements in security, documentation, and fixed some minor bugs. The DOI service now avoids returning entries that lack a URL, and it provides more reliable functionality for retrieving filenames from URLs. Security enhancements include a review of security alerts, the removal of stack trace outputs on the 404 page, and the elimination of default values from sensitive application properties. Additionally, a new AWS feature allows application properties to be optionally loaded from the AWS SSM Parameter Store. Our documentation revamp is complete and accessible on our [new documentation site](https://docs.eclipse-pass.org)! [Tickets Completed](https://github.com/eclipse-pass/main/milestone/27?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.12.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.12.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.12.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.12.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.12.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.12.0) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/1.12.0) ### Release v1.11.0 #### Date: September 30, 2024 Release Manager: Mark Patton, JHU This release fixed a few small bugs around retrieving and displaying manuscripts associated with a DOI, added the ability to display a failure message from a repository to a user and refreshed the Shibboleth setup of the local development environment. Work on enhancing our documentation and moving it to Gitbook is ongoing. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/26?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.11.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.11.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.11.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.11.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.11.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.11.0) ### Release v1.10.0 #### Date: August 28, 2024 Release Manager: Russ Poetker, JHU This release focused on a new Deposit Services repository integration, NIHMS Data Loader automation, and Release process and Documentation improvements. Deposit services has been enhanced to be able to deposit into InvenioRDM. This can also be tested locally with pass-docker being able to start a local instance of InvenioRDM. There is a new automation available in NIHMS Data Loader for refreshing the NIHMS API Token. We made a change to the pass-core/pass-support releases to align the maven repackage plugin configuration with its latest recommendations. As part of this change, the repackaged jar file is no longer deployed to Maven Central during release. The team continued work on overhauling and improving the existing documentation. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/24?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.10.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.10.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.10.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.10.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.10.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.10.0) ### Release v1.9.1 #### Date: August 1, 2024 Release Manager: Russ Poetker, JHU This release fixes a bug with the layout of some of the pages in the UI. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/25?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.9.1) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.9.1) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.9.1) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.9.1) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.9.1) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.9.1) ### Release v1.9.0 #### Date: July 31, 2024 Release Manager: Russ Poetker, JHU This release focused on improving deployment tests, addressing technical debt, and improving documentation. Deployment tests were updated to make deposits into downstream repositories optional. If a deployment test deposit is made into a downstream DSpace repository, it will be automatically deleted after the test completes. More deprecations in pass-ui were fixed which allowed pass-ui to be upgraded to Ember v5.8. The pass-ui module now uses Embroider and pnpm for building. The team continued work on overhauling the existing documentation. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/23?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.9.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.9.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.9.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.9.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.9.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.9.0) ### Release v1.8.0 #### Date: July 1, 2024 Release Manager: Jared Galanis, JHU This release focused on improving documentation, addressing technical debt, and fixing bugs. CSRF protection was added to several PASS components. An optional InvenioRDM instance was added to pass-docker. Many deprecations that were preventing an upgrade of pass-ui to Ember v5.x were addressed. The IDP configuration was reworked to be loaded more dynamically via a url instead of a file. The team advanced an overhaul of existing documentation, where many older sources of documentation were reviewed for accuracy, relevance and categorization, and some of which were subsequently synthesized into new forms of documentation. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/22?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.8.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.8.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.8.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.8.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.8.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.8.0) ### Release v1.7.0 #### Date: May 30, 2024 Release Manager: Russ Poetker, JHU This release focused on adding a GitHub workflow that will complete the release of all PASS components. The pass-core metadata schema service was changed as a first step in supporting InvenioRDM integration. There were several documentation tasks completed such as a Review Manual, Style Guide, and first round of reviews. There has been steps made to eventually add IaC for PASS. OpenTofu has been selected as the IaC tool; developing the terraform modules is in progress. A few smaller bugs were also fixed. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/21?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.7.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.7.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.7.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.7.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.7.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.7.0) ### Release v1.6.1 #### Date: May 7, 2024 Release Manager: Mark Patton, JHU This release fixes a bug which may prevent login and tweaks timeouts for monitoring NIHMS email. Tickets Completed: * [930](https://github.com/eclipse-pass/main/issues/930) * [974](https://github.com/eclipse-pass/main/issues/974) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.6.1) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.6.1) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.6.1) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.6.1) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.6.1) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.6.1) ### Release v1.6.0 #### Date: April 30, 2024 Release Manager: Mark Patton, JHU This release focused on simplifying authentication support, writing documentation to make collaboration easier, and automated testing against a live PASS instance. The pass-core component took over the responsibility for authentication and mediating access to pass-ui resources. This allowed us to remove the no longer needed pass-auth component. The deposit services now also cleanup after the new automated tests. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/20?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.6.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.6.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.6.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.6.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.6.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.6.0) ### Release v1.5.0 #### Date: March 28, 2024 Release Manager: Timothy Sanders, JHU This release focused on use cases for the planned Admin UI, documentation for the PASS Welcome Guide, and enhancements to the grant and nihms data loaders. We added updated parameters to the nihms loader for scheduled environments, and revised the nihms email processing. The grant loader had updates to its aggregation rules, CSV ingest, and extended test coverage. We simplified our CI/CD pipeline by creating a single action to deploy all PASS components to a specified environment. Began consolidating the authentication process to integrate Spring Security into pass-core, enhancing flexibility and security. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/19?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.5.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.5.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.5.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.5.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.5.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/1.5.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.5.0) ### Release v1.4.0 #### Date: February 28, 2024 Release Manager: Russ Poetker, JHU This release focused on updating dependency versions and enforcing clean dependency management in the PASS backend repositories. The required configuration architecture was simplified for the nihms and grant data loaders by making these Spring applications. We began working on a new documentation repository supported by GitBook, more to come on this in the near future. We improved the file delete action on the UI by deleting such files from the backend. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/18?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.4.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.4.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.4.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.4.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.4.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/1.4.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.4.0) ### Release v1.3.0 #### Date: January 31, 2024 Release Manager: Mark Patton, JHU This release focused on updating the interactions with NIHMS. A service was added to handle email messages from NIHMS about submission status. We made GitHub actions for Java snapshot and release builds consistent and more robust. We switched the grant loader to a CSV format which will make it easier to import grant data from other systems. For the UI, we improved the accessibility of the UI and the interaction with external links in the workflow. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/17?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.3.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.3.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.3.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.3.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.3.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/1.3.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.3.0) ### Release v1.2.0 #### Date: November 30, 2023 Release Manager: Timothy Sanders, JHU This release focused on increasing the security of PASS and making interactions with external services more robust, notably the interface with NIHMS. Parameterized queries were added to the grant loader to enhance security. We've made substantial upgrades to the NIHMS data transfer within our Deposit Services and enhancements to the NIHMS loader. Updates were made to the data model documentation and client-side pagination support has been added in the UI. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/16?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.2.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.2.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.2.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.2.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.2.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/1.2.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.2.0) ### Release v1.1.0 #### Date: October 26, 2023 Release Manager: Mark Patton, JHU This release focused on getting PASS ready for deployment in production. We did a great deal of testing of the user interface, backend services, and interactions with repositories. We found and fixed a large number of bugs. We significantly improved the performance of grant loading. We made accessibility improvements to the user interface. In addition, we added support for depositing to repositories without requiring a journal be entered by the user. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/15?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.1.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.1.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.1.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.1.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.1.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/1.1.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.1.0) ### Release v1.0.0 #### Date: September 29, 2023 Release Manager: Jared Galanis, JHU This release focused on setting up a PASS for production readiness. We resolved a large number of bugs in the user interface and the API / backend services. We added optimistic locking to Submission and Deposit entities to ensure more expected behavior when users edit a shared resource. We also did work on tooling for data migration and remediation. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/12?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.0.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.0.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.0.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.0.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.0.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/1.0.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.0.0) ### Release v0.9.0 #### Date: August 30, 2023 Release Manager: John Abrahams, JHU This release focused on setting up a staging environment for the PASS application. We deployed PASS to the new environment, integrated single sign-on and the data loaders, and fixed a number of bugs that were discovered in the refactored codebase. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/13?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/0.9.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/0.9.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/0.9.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/0.9.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/0.9.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/0.9.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/0.9.0) ### Release v0.8.0 #### Date: July 28, 2023 Release Manager: Mark Patton, JHU This release introduces updated Java implementations of pass-deposit-services. All of the major functionality of PASS has now been ported to the new framework. In addition more testing was added to the pass-core file service and support for the file service was added to pass-data-client. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/11?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/0.8.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/0.8.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/0.8.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/0.8.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/0.8.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/0.8.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/0.8.0) Release Manager: Mark Patton, JHU ### Release v0.7.0 #### Date: June 29, 2023 Release Managers: Christopher Shannon, JHU and Russell Poetker, JHU This release introduces updated java implementations of the pass-nihms-loader and pass-notification-services projects. This release also introduces support for sending submission and deposit JMS message from pass-core, adds access control to the file-service, removes pass-ui-public from pass-docker, and cleans up the SAML configuration in pass-auth. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/10?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/0.7.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/0.7.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/0.7.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/0.7.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/0.7.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/0.7.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/0.7.0) ### Release v0.6.0 #### Date: May 31, 2023 Release Manager: Jared Galanis, JHU This release introduces java implementations of the pass-journal-loader, pass-grant-loader and submission status service. This release also introduces support for user token authentication, updates to use of Java 17 in several repositories, converts pass-auth to TypeScript, integrates the user interface with the API for the policy service, introduces a simplified branding strategy along with default branding fallbacks to enable organization specific look and feel, and provides an action for publishing to an AWS SNS (Simple Notification Service) topic to facilitate deploying to AWS infrastructure. Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/0.6.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/0.6.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/0.6.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/0.6.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/0.6.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/0.6.0) * [pass-ui-public](https://github.com/eclipse-pass/pass-ui-public/releases/tag/0.6.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/0.6.0) ### Release v0.5.0 #### Date: April 27, 2023 Release Manager: Timothy Sanders, JHU This release introduces the Metadata Schema Service and the Policy Service API. The Metadata Schema Service provides JSON schemas for repository metadata requirements. The Policy Service API determines the policies applicable to a given Submission, as well as the repositories that a Submission must be deposited into. Release Automation has been expanded to include [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing) and [pass-docker](https://github.com/eclipse-pass/pass-docker). Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/0.5.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/0.5.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/0.5.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/0.5.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/0.5.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/0.5.0) * [pass-ui-public](https://github.com/eclipse-pass/pass-ui-public/releases/tag/0.5.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/0.5.0) ### Release v0.4.0 #### Date: March 30, 2023 Release Managers: John Abrahams, JHU and Christopher Shannon, JHU This release introduces a new user service and access control. The release also upgraded ember to the latest LTS Ember 4. * Updated Ember packages and 3rd party dependencies * Fixed styling post Ember 4 upgrade * Introduces User Service Integration * Introduces Access Control * Standardized the entrypoint of dockerfiles to point at an entrypoint.sh file Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/0.4.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/0.4.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/0.4.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/0.4.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/0.4.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/0.4.0) * [pass-ui-public](https://github.com/eclipse-pass/pass-ui-public/releases/tag/0.4.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/0.4.0) ### Release v0.3.0 #### Date: February 28, 2023 Release Manager: John Abrahams, JHU This release introduces a new file handling service for dealing with file uploads in PASS. Releases are now largely automated using GitHub workflows. * Release automations using GitHub workflows. Snapshot versions are published automatically and releases can be triggered manually in the GitHub UI * Add file API to pass-core for handling file related create, read, and delete operations * Add service to pass-core to look for publicly available manuscripts for a given DOI Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/0.3.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/0.3.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/0.3.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/0.3.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/0.3.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/0.3.0) * [pass-ui-public](https://github.com/eclipse-pass/pass-ui-public/releases/tag/0.3.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/0.3.0) ### Release v0.2.0 #### Date: January 18, 2023 Release Manager: Jim Martino, JHU Release 0.2.0 provides a major upgrade to the backend architecture of the PASS application. The Fedora Repository has been replaced with a completely new REST API built using Elide and backed by Postgres. This change allows the PASS API to be tailored more directly to the purposes of the PASS application, provides considerable performance enhancements, and reduces maintenance burden. The structure of the projects making up the PASS application have also been streamlined to simplify release and deployment procedures. These changes require updates to be made across the application, such as replacing all uses of the Fedora API within PASS with calls to the new API. For 0.2.0, this work is completed sufficiently to provide a demonstration of PASS application capabilities, but certain parts of the application are currently mocked. Full functionality will be restored in a future release. Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/0.2.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/0.2.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/0.2.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/0.2.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/0.2.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/v0.2.0) * [pass-ui-public](https://github.com/eclipse-pass/pass-ui-public/releases/tag/v0.2.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/0.2.0) Some major changes for v0.2.0 are: * Replacement of Fedora storage with Elide / postgres. * Creation of a pass-core repository which contains REST services previously in separate projects/images * Elimination of functionality written in Go in previous releases. These have either been implemented in Java or eliminated * Refactoring of authorization functionality in javascript closer to the UI ### Release v0.1.0 #### Date: August 3, 2022 Release Manager: John Abrahams, JHU This is the initial release of the Eclipse PASS codebase. The following changes were made to the code after completing the transition to Eclipse: Naming changes - updating code to transition to the Eclipse PASS name Version alignment - ensuring all project components utilize a consistent versioning scheme Data model alignment - ensuring all project components utilize the same version of the data model Adjustments to release process - updates to allow release of Java components to Maven Central with a new groupId Introduction of code style guide - ensuring code conforms to consistent guidelines Adjustments to testing methods - transitioning to the use of GitHub Actions for the execution of unit and integration testing Initial documentation - providing a starting point for development and deployment documentation Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/0.1.0) * [pass-ui-public](https://github.com/eclipse-pass/pass-ui-public/releases/tag/v0.1.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/v0.1.0) * [pass-ember-adapter](https://github.com/eclipse-pass/pass-ember-adapter/releases/tag/v0.1.0) * [pass-java-client](https://github.com/eclipse-pass/pass-java-client/releases/tag/0.1.0) * [pass-authz](https://github.com/eclipse-pass/pass-authz/releases/tag/0.1.0) * [pass-deposit-services](https://github.com/eclipse-pass/pass-deposit-services/releases/tag/0.1.0) * [pass-messaging-support](https://github.com/eclipse-pass/pass-messaging-support/releases/tag/0.1.0) * [pass-notification-services](https://github.com/eclipse-pass/pass-notification-services/releases/tag/0.1.0) * [pass-package-providers](https://github.com/eclipse-pass/pass-package-providers/releases/tag/0.1.0) * [pass-doi-service](https://github.com/eclipse-pass/pass-doi-service/releases/tag/0.1.0) * [pass-download-service](https://github.com/eclipse-pass/pass-download-service/releases/tag/0.1.0) * [pass-policy-service](https://github.com/eclipse-pass/pass-policy-service/releases/tag/0.1.0) * [pass-indexer-checker](https://github.com/eclipse-pass/pass-indexer-checker/releases/tag/0.1.0) * [pass-indexer](https://github.com/eclipse-pass/pass-indexer/releases/tag/0.1.0) * [pass-journal-loader](https://github.com/eclipse-pass/pass-journal-loader/releases/tag/0.1.0) * [pass-nihms-loader](https://github.com/eclipse-pass/pass-nihms-loader/releases/tag/0.1.0) * [pass-grant-loader](https://github.com/eclipse-pass/pass-grant-loader/releases/tag/0.1.0) * [pass-fcrepo-jsonld](https://github.com/eclipse-pass/pass-fcrepo-jsonld/releases/tag/0.1.1) * [pass-fcrepo-jms](https://github.com/eclipse-pass/pass-fcrepo-jms/releases/tag/0.1.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/0.1.0) These repositories were involved in the release, but do not have a `0.1.0` release: * [pass-fcrepo-module-auth-rbacl](https://github.com/eclipse-pass/pass-fcrepo-module-auth-rbacl) * [modeshape](https://github.com/eclipse-pass/modeshape) # Developer Documentation The PASS Developer Documentation encompasses all aspects of PASS required to understand and contribute to the PASS\ development. The main areas of PASS Development are PASS Core, PASS UI, Data Loaders, Deposit Services,\ Notification Services, and Acceptance Tests. All of these components work together to bring together a comprehensive\ system to disseminate research to their proper repositories. Each of these modules has a distinct role, such as managing\ data ingestion, handling user interactions, automating data transformations, and making deposits to their intended\ downstream repository. In all, these components interact to create an efficient workflow that supports research\ dissemination in various institutional and federal compliance scenarios. The Data Loaders handle data ingestion from external systems like PubMed Central, NIH, FIBI\ (JHU Grant Management System), transforming grant, journal, and manuscript submission data into a standardized format\ within PASS. Deposit Services then manage the packaging and transfer of these submissions to downstream repositories\ such as Pubmed Central and institutional repositories. Notification Services provide alerts and updates to relevant\ stakeholders based on submission workflows and events. This modular structure, built primarily using Java and Spring\ Boot, allows for flexibility and extensibility in accommodating different institutional requirements and third-party\ integrations. The developer documentation also details the technical setup, configuration, and deployment of each PASS component,\ including environment-specific configurations, database setup, authentication management, and integration with cloud\ services provided by AWS. PASS utilizes RESTful APIs for data access and manipulation, while its microservices\ communicate asynchronously using message queues to support scalable and distributed deployments. By following a modular\ and loosely-coupled architecture, PASS is able to evolve rapidly, incorporating new compliance requirements and\ enhancing repository deposit capabilities across academic and research institutions. **Table of Contents:** 1. [Use cases](/developer-documentation/use-cases) 2. [PASS Core](/developer-documentation/pass-core) 3. [PASS UI](/developer-documentation/pass-ui) 4. [Metadata Schema](https://github.com/eclipse-pass/pass-documentation/blob/main/developer-documentation/metadata-schema.md) 5. [Data Loaders](/developer-documentation/data-loaders) 6. [Deposit Services](/developer-documentation/deposit-service) 7. [Notification Services](/developer-documentation/notification-service) 8. [Pass Acceptance Testing](/developer-documentation/pass-acceptance-testing) 9. [PASS Docker](/developer-documentation/pass-docker) 10. [Release](/developer-documentation/release) # Use Cases ## Use Cases ## Direct submission * Principal Investigator (PI) users can review their grants and create submissions to designated repositories based on applicable policies. * A PI can see past submissions made to PubMed Central (PMC). * A submitter can create a submission that satisfies one or more grants' open access policies (PASS will determine which open access policies apply to a grant). * A submitter can create one submission that the system will deposit into an institutional repository and/or PubMed Central (potential support for more repositories in the future). ## Proxy submission * As an authenticated user of PASS, I want to be able to prepare a submission on behalf of someone else. * As an authenticated user of PASS, I want to be able to approve and submit submissions that were created on my behalf. ## Core features As a user, I can: * Login with my institutional ID. * See if I have submissions that need review on my dashboard. * View my grants. * View my submissions made through PASS. * View my submissions that I made to PMC in the past. * Associate a publication to a new submission by its DOI or manually enter its name and journal during the submission process. * Search for my publications journal, if entering its information manually during the submission process. * Create a submission on behalf of a grant PI. * Create a submission associated with one or more of my grants. * See all public / open access policies that apply to the grants associated with my submission during the submission process. * See the repositories during the submission process that the system will deposit into. * Enter metadata required by all target repositories during the submission process. * Add the manuscript file and zero or more supporting files to the submission during the submission process. * See the submissions status after I complete the submission. * See identifiers assigned to a successful submission by a repository. # PASS Core ## Summary This module is a Spring Boot and Elide application which provides HTTP APIs. PASS has a single page JavaScript user interface based on Ember. The UI interacts with pass-core through HTTP APIs. The HTTP APIs are JSON:API for CRUD and search on the PASS data model, a custom file API to handle binaries, a custom DOI API, a custom metadata schema API, a custom user service API, and a custom policy service API. The pass-core component runs these APIs, handles authentication, sends messages to queues, and mediates access to static HTTP resources. Several additional services run on the backend. [Data loaders](/developer-documentation/data-loaders) periodically update PASS with the latest information about institutional grants, journal metadata, and PubMed central publications. A deposit service turns submissions to PASS into deposits to repositories like DSpace and PubMed Central. A notification service sends email to users about events. The deposit and notification services monitory message queues. The deposit service also polls PASS for objects it needs to update. Elide provides a JSON:API based interface to the data model and persists the data model to a database. Both the UI and backend services interact with the data model using JSON:API. ## Knowledge Needed / Skills Inventory PASS Core covers a lot of technological and research domains. Basic understanding of grants, manuscript submission workflows, and the understanding of the technologies below will be beneficial to the development of Pass Core. * **Programming Languages** * [Java 17+](https://www.oracle.com/java/technologies/downloads/) * **Frameworks** * Knowledge of Spring Boot framework * Familiarity with [Elide](https://elide.io/) for data model management and JSON services * **API Development** * Creating and managing RESTful APIs * JSON specification ## Technologies Utilized * [Java 17+](https://www.oracle.com/java/technologies/downloads/) * [Spring Boot](https://spring.io/projects/spring-boot) * [Elide](https://elide.io/) * [Docker](https://www.docker.com/products/docker-desktop/) * [Amazon SQS](https://aws.amazon.com/sqs/) ## Technical Deep Dive ### Data Model The [data model](/developer-documentation/pass-core/model) holds all the information needed to associate users with grants and manage deposits to repositories. ### Building Java 17, Maven 3.8, and Docker are required. ```shell mvn clean install ``` This will produce an executabler jar `pass-core-main/target/pass-core-main--exec.jar` and a docker image `ghcr.io/eclipse-pass/pass-core-main`. #### Running the local build After you have run `mvn clean install`, execute the following command from the `pass-core-main` directory: ```shell java -Dspring.config.import=file:./src/test/resources/application-test.yml -jar target/pass-core-main--exec.jar ``` This command will use the configuration defined in the `pass-core-main/src/test/resources/application-test.yml` file. **This configuration should not be used in production, it is only meant for testing purposes.** You can verify it is running by making a request like: ```shell curl -u backend:moo localhost:8080/data/grant ``` #### Running with Docker Run `mvn clean install`. Then go to the [pass-docker](https://github.com/eclipse-pass/pass-docker) repository and following the instructions for starting a local environment. ### Configuration The application is configured by its `pass-core-main/src/main/resources/application.yaml` which in turn references a number of environment variables. By default, pass-core-main will run with a typical production configuration. In order to run the default configuration, the environment variables below must be set with appropriate values for your environment. Environment variables: | Environment Variable | Default Value | Description | | ------------------------------------------------ | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `PASS_CORE_APP_LOCATION` | classpath:app/ | Location where requests are resolved | | `PASS_CORE_DATABASE_URL` | | Connection URL to database | | `PASS_CORE_DATABASE_USERNAME` | | Username for database login | | `PASS_CORE_DATABASE_PASSWORD` | | Password for database login | | `PASS_CORE_PORT` | | The port to expose for pass-core API | | `PASS_CORE_LOG_DIR` | ${java.io.tmpdir}/pass-core | Path to log directory | | `PASS_CORE_USER` | | Default user name for pass-core | | `PASS_CORE_PASSWORD` | | Default user password for pass-core | | `PASS_CORE_USE_SQS` | true | Flag to use AWS SQS for messaging | | `PASS_CORE_EMBED_JMS_BROKER` | false | Flag to use Embedded ActiveMQ for messaging | | `PASS_CORE_SUBMISSION_QUEUE` | pass-submission | Name of submission queue | | `PASS_CORE_DEPOSIT_QUEUE` | pass-deposit | Name of deposit queue | | `PASS_CORE_SUBMISSION_EVENT_QUEUE` | pass-submission-event | Name of submission event queue | | `PASS_CORE_SP_ID` | | SAML SP ID [SAML configuration](#saml-configuration) | | `PASS_CORE_SP_ACS` | | SAML SP ACS [SAML configuration](#saml-configuration) | | `PASS_CORE_SP_KEY` | | Location of SAML SP private key pem file [SAML configuration](#saml-configuration) | | `PASS_CORE_SP_CERT` | | Location of SAML SP public certificate pem file [SAML configuration](#saml-configuration) | | `PASS_CORE_IDP_METADATA` | | Location of SAML IDM Metadata file [SAML configuration](#saml-configuration) | | `PASS_CORE_APP_CSP` | | The Content Security Policy definition | | `PASS_CORE_DEAULT_LOGIN_SUCCESS` | | Path to redirect to after login success [SAML configuration](#saml-configuration) | | `PASS_CORE_LOGIN_PROCESSING_PATH` | | Path to handle login from SAML IDP [SAML configuration](#saml-configuration) | | `PASS_CORE_LOGOUT_SUCCESS` | | Path to redirect to after SAML logout [SAML configuration](#saml-configuration) | | `PASS_CORE_LOGOUT_DELETE_COOKIES` | | Name of cookies to delete as part of SAML logout [SAML configuration](#saml-configuration) | | `PASS_CORE_USERTOKEN_KEY` | | If not present, one is generated. See the [user service](https://github.com/eclipse-pass/pass-documentation/blob/main/developer-documentation/pass-core/pass-core-user-service/README.md) for how to create manually. | | `PASS_CORE_JAVA_OPTS` | | Used by the Docker image to pass arguments to Java. | | `PASS_CORE_BASE_URL` | | Used when services send URLs to the client such as relationship links. | | `PASS_CORE_FILE_SERVICE_TYPE` | FILE\_SYSTEM | The type of File Service, FILE\_SYSTEM or S3 | | `PASS_CORE_FILE_SERVICE_ROOT_DIR` | | Path to File Service root directory | | `PASS_CORE_S3_BUCKET_NAME` | pass-core-file | If File Service is S3, the S3 bucket name | | `PASS_CORE_S3_REPO_PREFIX` | pass-core-file | If File Service is S3, the prefix of S3 keys in the bucket | | `PASS_CORE_POLICY_INSTITUTION` | | Name of the institution | | `PASS_CORE_POLICY_INSTITUTIONAL_POLICY_TITLE` | | Title of the institutional policy | | `PASS_CORE_POLICY_INSTITUTIONAL_REPOSITORY_NAME` | | Name of institutional repository | The liquibase changelog located `pass-core-main/src/main/resources/db/changelog/changelog.yaml` will create the pass-core database schema if needed. If `PASS_CORE_USE_SQS` is `true`, then pass-core will attempt to connect to Amazon SQS. For testing purposes, you can set `AWS_REGION`, `AWS_ACCESS_KEY_ID`, and `AWS_SECRET_ACCESS_KEY` for connecting to AWS resources. In production, AWS IAM Service Roles should be used. Otherwise, a connection to an ActiveMQ broker can be configured by setting `SPRING_ACTIVEMQ_BROKER_URL`. If `PASS_CORE_EMBED_JMS_BROKER` is true, then an embedded ActiveMQ broker will be started\ using that url. This can be useful to set tcp transport for connecting containers in a docker environment. The default is an embedded broker using vm transport. **Note you can quickly start pass-core locally for testing purposes following the instructions in** [**Running local build**](#running-local-build) **section.** ### Access control SAML 2.0 and HTTP basic authentication are supported. An authenticated user is either authorized with a `BACKEND` or `SUBMITTER` role. A user that does a SAML login is mapped to a PASS user using locator ids. The provided SAML properties of the user\ are interpreted using the spring property `pass.auth.attribute-map`. The user is assigned the `SUBMITTER` role. There is a single `BACKEND` user specified who logs in using HTTP basic. The `BACKEND` role can do everything. The `SUBMITTER` role is restricted to creating and modifying certain objects in the data model.\ The `SUBMITTER` has full access to all other services. Details are available in the [Authentication and Authorization section](/developer-documentation/pass-core/authentication-authorization). ### SAML configuration The `PASS_CORE_SP_KEY` and `PASS_CORE_SP_CERT` environment variables set the location of the keys used by pass-core to encrypt SAML communication.\ Use `PASS_CORE_SP_ID` to set the identifier of the pass-core SP, `PASS_CORE_IDP_METADATA` to set the location where IDP metadata can be retrieved,`PASS_CORE_SP_ACS` for the Assertion Consumer Service of the SP and `PASS_CORE_LOGIN_PROCESSING_PATH` to set the path for handling login from the IDP.\ Note that `PASS_CORE_SP_ACS` is a URL which must match the path specified in `PASS_CORE_LOGIN_PROCESSING_PATH`. The `application-test.yml` configuration is are set such that the integration tests can run against a [SimpleSAMLphp based IDP](https://github.com/kenchan0130/docker-simplesamlphp/) using resources included in `saml2/`. These defaults should not be used in production. The image can be run with: ```shell docker run --name=idp -p 8090:8080 -e SIMPLESAMLPHP_SP_ENTITY_ID=https://sp.pass/shibboleth -e SIMPLESAMLPHP_SP_ASSERTION_CONSUMER_SERVICE=http://localhost:8080/login/saml2/sso/pass -e SIMPLESAMLPHP_IDP_BASE_URL=http://localhost:8090/ -v ./pass-core/pass-core/main/src/test/resources/saml2/authsources.php:/var/www/simplesamlphp/config/authsources.php -d kenchan0130/simplesamlphp ``` Note the volume mount which is set the user information appropriately for PASS. ### CSRF protection Requests which have side effects (not a GET, HEAD, or OPTIONS and any request to /doi) are protected from CSRF through the use of a token. The client must provide a cookie XSRF-TOKEN and set a header X-XSRF-TOKEN to the same value. Clients can use any value they want. Browser clients will have the cookie value set by responses and so must first make a non-protected request. ### APIs #### App `/app/` The PASS application is available at `/app/` and `/` is redirected to `/app/`. Requests are resolved against the location given by the environment variable `PASS_CORE_APP_LOCATION`. If a request cannot be resolved, then `/app/index.html` will be returned. This allows the user interface to handle paths which may not resolve to files. #### User `/user/` The [user API](/developer-documentation/pass-core/api/user) provides information about the logged in user. #### DOI `/doi/` The [DOI API](/developer-documentation/pass-core/api/doi) provides the ability to interact with DOIs. #### File `/file/` The [file API](/developer-documentation/pass-core/api/file) provides a mechanism to persist files. #### Policy `/policy/` The [policy API](/developer-documentation/pass-core/api/policy) indicates what repositories are publication should be pushed to. #### JSON API JSON API is deployed at the `/data/` endpoint. All of our data model is available, just divided into attributes and relationships. Note that identifiers are now integers, not URIs.\ See the [Elide docs](https://elide.io/pages/guide/v6/10-jsonapi.html) for information on how Elide provides support for filtering and sorting. See the `/swagger/` endpoint for auto-generated documentation. You can directly make request with the UI and see what happens. Note when doing a POST to create an object, be sure to edit the type field to have the correct object type and delete the id field to have the id auto-generated. ### Examples #### Creating a RepositoryCopy ```shell curl -v -u backend:moo -H "X-XSRF-TOKEN:token" -H "Cookie:XSRF-TOKEN=token" -X POST "http://localhost:8080/data/repositoryCopy" -H "accept: application/vnd.api+json" -H "Content-Type: application/vnd.api+json" -d @rc1.json ``` *rc1.json:* ```json { "data": { "type": "repositoryCopy", "attributes": { "accessUrl": "http://example.com/path", "copyStatus": "accepted" } } } ``` #### Patch a Journal Add a publisher object to the publisher relationship in a journal. Note that both the journal and publisher objects must already exist. ```shell curl -u backend:moo -H "X-XSRF-TOKEN:token" -H "Cookie:XSRF-TOKEN=token" -X PATCH "http://localhost:8080/data/journal/1" -H "accept: application/vnd.api+json" -H "Content-Type: application/vnd.api+json" -d @patch.json ``` *patch.json:* ```json { "data": { "type": "journal", "id": "1", "relationships": { "publisher": { "data": { "id": "2", "type": "publisher" } } } } } ``` ### Messages Messages are JSON objects emitted to a JMS broker as text messages. The different types of messages are sent to different queues specified\ by the indicated by the environment variables `PASS_CORE_SUBMISSION_QUEUE`, `PASS_CORE_SUBMISSION_EVENT_QUEUE`, and `PASS_CORE_DEPOSIT_QUEUE`. When a Submission is created or modified and the submitted field is true, then a SubmissionReady event is emitted.\ The id of the Submission will be set in the `submission` field of the message. When a SubmissionEvent is created, then the a SubmissionEvent message will be sent.\ The id of the SubmissionEvent will be set in the `submission-event` field of the message. If the `eventType` field is `APPROVAL_REQUESTED_NEWUSER`,\ then an `approval-link` field will be set in the field of the message with a link to be sent to a user. When a Deposit is created or modified, then a DepositStatus event is emitted.\ The id of the Deposit will be set in the `deposit` field of the message. **Example messages:** ```json { "type": "SubmissionReady", "submission": "1" } ``` ```json { "type": "DepositStatus", "deposit": "1" } ``` ```json { "type": "SubmissionEvent", "submission-event": "1", "approval-link": "http://example.com/passui?userToken=xxxx" } ``` ### Debugging problems To get more information, try changing the logging levels set in `pass-core-main/src/main/resources/logback-spring.xml`.\ You might also try setting properties like `-Dlogging.level.org.eclipse.pass=DEBUG`. The [Elide Docs](https://elide.io/pages/guide/v6/12-audit.html) provide more information on logging and debugging. ## Next Steps / Institution Configuration ### Environmental Variable Setup PASS is designed to be flexible and can be easily configured using the environment variables in this section and the environment variables mentioned in the [API section](https://github.com/eclipse-pass/pass-documentation/blob/main/developer-documentation/pass-core/api/README.md) as well. Key variables to consider include: * PASS Admin and URLs: ``` PASS_CORE_BASE_URL=[your-base-url] PASS_CORE_USER=[your-username] PASS_CORE_PASSWORD=[your-password] PASS_CORE_APP_LOCATION=[url-of-front-end] ``` * Database Configuration: Set up the database connection details for PostgreSQL: ``` PASS_CORE_DATABASE_URL=jdbc:postgresql://[your-database-host]:5432/[database-name] PASS_CORE_DATABASE_USERNAME=[your-username] PASS_CORE_DATABASE_PASSWORD=[your-password] ``` * SAML Authentication: Provide the institution-specific values for SAML configurations: ``` PASS_CORE_IDP_METADATA=classpath:saml2/[institution-idp-metadata].xml PASS_CORE_SP_ID=https://[your-domain]/shibboleth PASS_CORE_SP_CERT=classpath:saml2/[institution-sp-cert].pem PASS_CORE_SP_KEY=classpath:saml2/[institution-sp-key].pem PASS_CORE_SP_ACS=http://[your-domain]:8080/login/saml2/sso/pass ``` * AWS Configuration (if using SQS or S3): ``` AWS_REGION=[aws-region] AWS_ACCESS_KEY_ID=[aws-access-key] AWS_SECRET_ACCESS_KEY=[aws-secret-key] PASS_CORE_USE_SQS=true # If using S3 bucket for the file service PASS_CORE_FILE_SERVICE_TYPE=S3 PASS_CORE_S3_BUCKET_NAME=[your-bucket-name] PASS_CORE_S3_REPO_PREFIX=[your-prefix-name] ``` Review and adjust the other environment variables (e.g., queues, ports, CSP policies) as necessary to suit the institution's security and operational policies. ## Database Setup If using a new PostgreSQL instance, ensure the schema is correctly initialized. The liquibase changelog `(src/main/resources/db/changelog/changelog.yaml)` will handle the schema setup automatically. Verify that: * The schema is created correctly. * Any institution-specific schema changes or extensions are applied. ## Custom Policy Configuration Institutions may have specific deposit and submission policies. Configure these by following the instructions in the [Policy API section](/developer-documentation/pass-core/api/policy). This will ensure that the appropriate rules and repositories are applied based on your institutional guidelines. # Authentication & Authorization PASS handles authentication through `pass-core`, which initiates SAML exchanges with a Shibboleth-supporting identity provider to authenticate users and establish sessions based on specific attributes. Authorization is role-based, assigning users either a `SUBMITTER` or `BACKEND` role, which dictates their permissions for accessing and modifying resources within the system. ## Authentication ### User Interface Authentication Authentication for the user interface occurs through the use of an authentication service provider (SP), [pass-core](https://github.com/eclipse-pass/pass-core). `pass-core` is configured to initiate a SAML exchange with a known identity provider (IDP) that supports [Shibboleth](https://shibboleth.atlassian.net/wiki/spaces/CONCEPT/overview). Although `pass-core` itself is not a Shibboleth service provider specifically, it is a generalized SAML service provider that can handle specific Shibboleth interactions with an IDP. In response to a valid `authn` assertion against an IDP, `pass-core` expects to receive and validate a Shibboleth SAML assertion against its assertion consumer service (ACS) URL. This assertion is expected to contain the following Shibboleth attributes: ``` 'urn:oid:2.16.840.1.113730.3.1.241': 'Display name' 'urn:oid:1.3.6.1.4.1.5923.1.1.1.9': 'Scoped affiliation' 'urn:oid:0.9.2342.19200300.100.1.3': 'Email' 'urn:oid:2.16.840.1.113730.3.1.3': 'Employee id' 'urn:oid:1.3.6.1.4.1.5923.1.1.1.6': 'eduPersonPrincipalName' 'urn:oid:2.5.4.42': 'Given name' 'urn:oid:2.5.4.4': 'Surname' 'urn:oid:1.3.6.1.4.1.5923.1.1.1.13': 'Unique id' ``` These Shibboleth attributes are used to locate a user in `pass-core` and set up a user object on the session. `pass-core`, establishes a server side session and delivers a http-only cookie to the browser client which [`pass-ui`](https://github.com/eclipse-pass/pass-ui/) will use to establish a client side session in the user interface. This http-only cookie is delivered back to `pass-core` by the user interface with every request. This series of interactions is depicted as follows:
Authentication Interactions Diagram

Authentication Interactions Diagram

### REST API Authentication Every request to the [REST API](https://github.com/eclipse-pass/pass-core) must be authenticated. If it is not authenticated, it is denied. Requests to the API come from two types of clients, backend services and users. Requests from users must have already been authenticated with Shibboleth and have the headers specified above. If a request contains Shibboleth headers, it is considered trusted, authentication succeeds, it is associated with a user, and given the SUBMITTER role. If the user does not exist, it is created. If the user does exist, it is updated to reflect the information in the headers. If the request does not contain the Shibboleth headers, it undergoes HTTP basic authentication. There is one HTTP basic user defined with the `BACKEND` role for the backend services. Mapping from Shibboleth attributes to PASS users: * displayName: Display name * email: Email * firstName: Given name * lastName: Surname * username: Eppn * affiliations: DOMAIN, all values * locatorIds: UNIQUE ID, INSTITUTIONAL\_ID, EMPLOYEE\_ID * role: SUBMITTER The `DOMAIN` is the value of the Eppn attribute after `@`. The `UNIQUE_ID` is `DOMAIN:unique-id:` joined to the value of the unique id attribute before `@`. The `INSTITUTIONAL_ID` is `DOMAIN:eppn` joined to the value of the Eppn attribute before the `@`. The `EMPLOYEE_ID` is `DOMAIN:employeeid` joined to the value of the Employee id value. The `locatorIds` are used to find an existing user in the system. If any of the `locatorIds` match an existing user, the user is considered to match. ### Example mapping Shibboleth attributes: * `Eppn`: * `Display name`: Sally M. Submitter * `Mail`: * `Given name`: Sally * `Surnamen`: Submitter * `Affiliation`: * `Employee id`: 02342342 * `Unique id`: Resulting User: * `affiliation`: , johnshopkins.edu * `displayName`: Sally M. Submitter * `email`: * `firstName`: Sally * `lastName`: Submitter * `locatorIds`: johnshopkins.edu:unique-id:sms2323, johnshopkins.edu:eppn:sallysubmitter, johnshopkins.edu:employeeid:02342342 * `roles`: SUBMITTER * `username`: ## Authorization Requests either have a `SUBMITTER` or `BACKEND` role. The `BACKEND` can do everything. The `SUBMITTER` is restricted to creating and modifying certain objects in the data model. The `SUBMITTER` has full access to all other services. Object permissions: | Type | Create | Read | Update | Delete | | --------------- | -------------------------- | ---- | -------------------------- | -------------------------- | | Submission | BACKEND or SUBMITTER | any | BACKEND or owns submission | BACKEND or owns submission | | SubmissionEvent | BACKEND or owns submission | any | BACKEND | BACKEND | | File | BACKEND or owns submission | any | BACKEND or owns submission | BACKEND or owns submission | | Publication | BACKEND or owns submission | any | BACKEND or owns submission | BACKEND or owns submission | | \* | BACKEND | any | BACKEND | BACKEND | The permissions are all role based, except "owns submission". By "owns submission" means that the user is the submitter or a preparer on a submission associated with the object. A submitter is the target of the submitter relationship on a Submission. A preparer is the target of the preparers relationship on a `Submission`. `SubmissionEvent` and `File` are associated with a submission through a submission relationship. The intent is to make sure submitters can only modify submissions which they have created or are explicitly allowed to help on. # API # DOI API The DOI API has two services. One service returns the corresponding PASS journal for the article identified by a DOI as well as the article's CrossRef metadata. The other service returns information about a manuscript from Unpaywall. Both services accept the DOI of a journal article as a query parameter with a name of `doi`. The DOI should be formatted like `10.1234/ ...`. If a DOI is of a longer URL form containing the string `doi.org/`, then we truncate the DOI to take everything after this substring. If the DOI is not valid, a `400 Bad Request` status code is returned. On success a `200 OK` is returned. If there is an error the response will be a JSON object with an error key containing a message. The services will look for an environment variable called `PASS_DOI_SERVICE_MAILTO` to specify a value on the User-Agent header on the Crossref and Unpaywall requests. ## `/doi/journal` This service uses the Crossref API to get information about the article and its journal. We then check to see if there is a `Journal` object in PASS for this journal. If not we create one. The service then returns to the caller a JSON object containing the `journal-id` of the PASS journal, and a `crossref` object representing the data returned to the service as a result of the Crossref call. See the [Crossref docs](https://www.crossref.org/documentation/) for information on the Crossref metadata returned. If a request is already being processed for the given DOI, `429 Too Many Requests` is returned. ### Example Running this command: ```shell curl -u BACKEND_USER:BACKEND_PASS -H "X-XSRF-TOKEN:token" -H "Cookie:XSRF-TOKEN=token" localhost:8080/doi/journal?doi=DOI ``` Will return this JSON: ```JSON {"journal-id":"72","crossref": {}} ``` ## `/doi/manuscript` This service uses Unpaywall API to get information about the corresponding locations on the web for manuscript PDFs related to the article referenced by the DOI. Ultimately, we want a user of PASS to be informed of open access manuscripts that already exist on the web and be able to use those copies in their PASS submission, instead of having to manually upload the manuscript file(s). The external service URLs are configured by the `XREF_BASEURI` and `UNPAYWALL_BASEURI` environment variables which default to `https://api.crossref.org/v1/works/` and `https://api.unpaywall.org/v2/` respectively. On success a JSON object is returned with a key of manuscripts containing an array of manuscripts available to download. Each entry in the array is a JSON object. Manuscript entry: * `url`: The url to download the manuscript. * `repositoryLabel`: The repository containing the manuscript, PubMed Central. * `type`: The mime type of the manuscript, application/pdf. * `source`: The source used to find the manuscript, Unpaywall. * `name`: The file name of the manuscript. * `isBest`: A boolean indicating whether this is the best entry to use ## Example Running this command: ```shell curl -u BACKEND_USER:BACKEND_PASS -H "X-XSRF-TOKEN:token" -H "Cookie:XSRF-TOKEN=token" localhost:8080/doi/manuscript?doi=DOI ``` Will return this JSON: ```JSON "manuscripts": [ { "url": null, "repositoryLabel": "PubMed Central - Europe PMC", "type": "application/pdf", "source": "Unpaywall", "name": "test.pdf", "isBest": false }, { "url": "http://pdfs.semanticscholar.org/good.pdf", "repositoryLabel": null, "type": "application/pdf", "source": "Unpaywall", "name": "good.pdf", "isBest": true } ] ``` # File API The file is a RESTful service that provides the ability to upload, download, and delete files to a configured persistence store. The service is currently designed to persist to a filesystem or S3 compatible storage. ## Configuration The service is configured via environment variables. The service by default will use a filesystem based persistence store and does not require any additional configuration. If the variable `PASS_CORE_FILE_SERVICE_ROOT_DIR` does not have any value, the File Service will default to the system temp folder and create a temporary root folder of a random value in the system temp. The variable `PASS_CORE_FILE_SERVICE_ROOT_DIR` is used by both the `FILE_SYSTEM` and `S3` service types. It is the root directory where temporary files are stored before being persisted to the configured persistence store as specified by `PASS_CORE_FILE_SERVICE_TYPE`. If using `FILE_SYSTEM` as the persistence store, `PASS_CORE_FILE_SERVICE_ROOT_DIR` is also the root directory for file persistence. The value for the `PASS_CORE_FILE_SERVICE_ROOT_DIR` cannot be a S3 bucket, and it must be a valid path on the local filesystem. The following environment variables are available for configuring the service: * `PASS_CORE_FILE_SERVICE_TYPE=FILE_SYSTEM` * Currently supports \[`FILE_SYSTEM` | `S3`] * `PASS_CORE_FILE_SERVICE_ROOT_DIR=/path/to/root/dir` * The root directory of the service that is used to support file uploads and downloads and the root directory for file persistence if using `FILE_SYSTEM` as the persistence store. * Default example: system\_tmp/17318424270250529523 * `PASS_CORE_S3_BUCKET_NAME=bucket-test-name` * The name of the S3 bucket to use for file persistence if using S3 as the persistence store. * `PASS_CORE_S3_REPO_PREFIX=s3-repo-prefix` * `PASS_CORE_S3_ENDPOINT=http://localhost:9090` * If using a custom endpoint for S3, this value should be set to the endpoint URL. ## HTTP Error Responses The service will return the following HTTP error responses: * 400 - Bad Request * This error is returned when a file is empty or missing. It will also handle exceptions that are thrown by the OCFL library. * 404 - Not Found * This is returned when performing a GET/DELETE and the fileId is invalid * 500 - Internal Server Error * This error is returned when an unexpected error occurs in the service. ## Usage Examples ### Upload a file ```shell curl -u BACKEND_USER:BACKEND_PASS -H "X-XSRF-TOKEN:token" -H "Cookie:XSRF-TOKEN=token" -X POST "http://localhost:8080/file" -H "accept: application/json" -H "Content-Type: multipart/form-data" -F "file=@/path/to/file" ``` ### Download a file ```shell curl -u BACKEND_USER:BACKEND_PASS -X GET "http://localhost:8080/file/{uuid}/{origFileName}" -H "accept: application/octet-stream" --output /path/to/file" ``` ### Delete a file ```shell curl -u BACKEND_USER:BACKEND_PASS -H "X-XSRF-TOKEN:token" -H "Cookie:XSRF-TOKEN=token" -X DELETE "http://localhost:8080/file/{fileId}/{origFileName}" -H "accept: application/json" ``` # Policy API Contains the PASS policy service, which provides an HTTP API for determining the policies applicable to a given Submission, as well as the repositories that must be deposited into in order to comply with the applicable policies. ## Configuration Configuration is achieved via the following environment variables: * `PASS_POLICY_INSTITUTION`: This is the institution as it appears on User.affiliations for every user in the institution: e.g. "johnshopkins.edu" * `PASS_POLICY_INSTITUTIONAL_POLICY_TITLE`: The value of Policy.title on the institution's Policy object * `PASS_POLICY_INSTITUTIONAL_REPOSITORY_NAME`: The value of Repository.name on the intstitution's IR Repository object ## Policy Service The `/policy/policies` endpoint determines the set of policies that are applicable to a given submission. Note: The results may be dependent on *who* submits the request. For example, if someone from JHU invokes the policies endpoint, a general "policy for JHU employees" may be included in the results. ### Policies Request `GET /policy/policies?submission=${SUBMISSION_ID}` ### Policies Response The response is a list of IDs to Policy resources, decorated with a `type` property. A type of `institution` indicates that this is a institutional policy, a type of `funder` indicates that the policy is describes the requirements of a funder. ```JSON [ { "id": "3", "type": "funder" }, { "id": "22", "type": "institution" } ] ``` ## Repositories Service The `/policy/repositories` endpoint, for a given submission, calculates the repositories that may be deposited into in order to satisfy any applicable policies for that submission. ### Repositories Request GET `/policy-service/repositories?submission=${SUBMISSION_ID}` or, with urlencoded (with encoded submission=${SUBMISSION\_ID}) as the body: ### Repositories Response The response is an application/json document that lists repositories sorted into required and optional buckets. Required repositories must be deposited to while optional repositories may be deposited to. ```JSON { "required": [ { "url": "1", "selected": false }, { "url": "2", "selected": false }, { "url": "3", "selected": false }, { "url": "4", "selected": false }, { "url": "5", "selected": false } ], "optional": [ { "url": "6", "selected": true } ] } ``` Repositories contained in the above list are JSON objects containing the following fields: * `url`: the URL to the repository resource in Fedora * `selected`: optional field. Specifies if the repository should be selected by default in the UI or not. # User API The user API provides information about the currently authenticated user. The endpoint is `/user/whoami` which will return a JSON object on a GET request. The JSON object tells the client which PASS object represents the authenticated user. Example result: ```JSON { "id": "1234", "type": "user", "uri": "http://localhost:8080/data/user/1234" } ``` The parameter `userToken` can be used to provide a user token. The user token must be associated with a submission and an email address. A user token is an encrypted tuple consisting of a PASS resource URI and a reference URI. A request with a user token will return as normal, but have the side effect of setting the submitter of the submission to the current user. In order for that to happen the submitterEmail address field on the submission must match the email address of the token. Then environment variable `PASS_CORE_USERTOKEN_KEY` provides the key used to encrypt and decrypt user tokens. If it is not provided or empty, user token support is disabled. To generate a key run the class, `org.eclipse.pass.usertoken.KeyGenerator`. One way to do that is from `pass-core-usertoken` to do `mvn compile exec:java -Dexec.mainClass="org.eclipse.pass.usertoken.KeyGenerator"`. The purpose of user tokens is to handle the case of a request for a user who has not logged into PASS and so does not have a User object to handle a submission. This is part of the proxy functionality. # Model The PASS data model is represented using [JSON API](https://jsonapi.org/). ## Model Objects * [Deposit](https://github.com/eclipse-pass/pass-documentation/blob/main/developer-documentation/pass-core/model/model/Deposit.md): An attempt to push a publication to a repository. * [File](https://github.com/eclipse-pass/pass-documentation/blob/main/developer-documentation/pass-core/model/model/File.md): File being sent to a repository. * [Funder](https://github.com/eclipse-pass/pass-documentation/blob/main/developer-documentation/pass-core/model/model/Funder.md): The sponsor of a grant. * [Grant](https://github.com/eclipse-pass/pass-documentation/blob/main/developer-documentation/pass-core/model/model/Grant.md): Associates an award at the insitution with funders and principal investigators. * [Journal](https://github.com/eclipse-pass/pass-documentation/blob/main/developer-documentation/pass-core/model/model/Journal.md): The journal of a publication * [Policy](https://github.com/eclipse-pass/pass-documentation/blob/main/developer-documentation/pass-core/model/model/Policy.md): The institutional requirements to publish to certain repositories. * [Publication](https://github.com/eclipse-pass/pass-documentation/blob/main/developer-documentation/pass-core/model/model/Publication.md): The publication being sent to a repository * [Repository](https://github.com/eclipse-pass/pass-documentation/blob/main/developer-documentation/pass-core/model/model/Repository.md): The destination of a deposit * [RepositoryCopy](https://github.com/eclipse-pass/pass-documentation/blob/main/developer-documentation/pass-core/model/model/RepositoryCopy.md): A publication in a repository. * [Submission](https://github.com/eclipse-pass/pass-documentation/blob/main/developer-documentation/pass-core/model/model/Submission.md): The submission of a publication to a set of repositories. * [SubmissionEvent](https://github.com/eclipse-pass/pass-documentation/blob/main/developer-documentation/pass-core/model/model/SubmissionEvent.md): An event performed by a user on a submission. * [User](https://github.com/eclipse-pass/pass-documentation/blob/main/developer-documentation/pass-core/model/model/User.md): A PASS user. ## Model Diagram
Data Model Diagram

Data Model Diagram

## Notes ### Identifiers An object is uniquely identified by a tuple consisting of its id attribute and its type. ### DateTime attributes DateTime attributes are strings formatted as per the Java DateTimeFormatter with pattern `yyyy-MM-dd'T'HH:mm:ss.SSSX`. # Deposit A [Submission](/developer-documentation/pass-core/model/submission) can have multiple Deposits, each to a different [Repository](/developer-documentation/pass-core/model/repository). This entity describes the interaction of PASS with a target [Repository](/developer-documentation/pass-core/model/repository) for an individual [Submission](/developer-documentation/pass-core/model/submission) with the purpose of satisfying one or more [Policies](/developer-documentation/pass-core/model/policy). | Attribute | Type | Description | | ---------------- | ------ | ------------------------------------------------------------------------------------------------------------------------- | | id\* | String | Autogenerated identifier of object. | | depositStatusRef | String | A URL or some kind of reference that can be dereferenced, entity body parsed, and used to determine the status of Deposit | | depositStatus\* | String | Status of deposit ([*see list below*](#deposit-status-options)) | | statusMessage | String | A human readable messabe about the deposit | | version\* | Long | Version number that increments on updates. Used to check update requests and ensure consistency. | | Relationship | Type | Target | Description | | -------------- | ------ | -------------------------------------------------------------------------- | ------------------------------------ | | submission\* | To One | [Submission](/developer-documentation/pass-core/model/submission) | Submission this Deposit is a part of | | repository\* | To One | [Repository](/developer-documentation/pass-core/model/repository) | Repository being deposited to | | repositoryCopy | To One | [Repository Copy](/developer-documentation/pass-core/model/repositorycopy) | Repository Copy for this Deposit | \*required ## Deposit status options These are the possible statuses for a Deposit in the order they could occur. Note that not all repositories will go through every status. Intermediate statusA Deposit with an *intermediate* status indicates that the processing of the Deposit is not yet complete. At some indeterminate point in the future, the status *may* be updated to a *terminal* state.Terminal statusA Deposit with a *terminal* status indicates that the processing of the Deposit is complete. | Value | State | Description | | --------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | submitted | Intermediate | PASS has sent a package to the target [Repository](/developer-documentation/pass-core/model/repository) and is waiting for an update on the status | | rejected | Terminal | The target [Repository](/developer-documentation/pass-core/model/repository) has rejected the Deposit | | failed | Intermediate | A failure occurred while performing the deposit, it may be re-tried later. | | accepted | Terminal | The target [Repository](/developer-documentation/pass-core/model/repository) has accepted the [Files](/developer-documentation/pass-core/model/file) into the repository and they are pending publication if not published already | # File Files are associated with a [Submissions](/developer-documentation/pass-core/model/submission) to be used to form [Deposits](/developer-documentation/pass-core/model/deposit) into [Repositories](/developer-documentation/pass-core/model/repository) | Attribute | Type | Description | | ----------- | ------ | ------------------------------------------------------------------------------------- | | id\* | String | Autogenerated identifier of object | | name\* | String | File name, defaults to filesystem name | | uri\* | String | Relative URI to the file servive which will return the bytestream | | description | String | Description of file provided by [User](/developer-documentation/pass-core/model/user) | | fileRole | String | Role of the file ([*see list below*](#file-role-options)) | | mimeType | String | Mime-type of file | | Relationship | Type | Target | Description | | ------------ | ------ | ----------------------------------------------------------------- | -------------------------------- | | submission\* | To One | [Submission](/developer-documentation/pass-core/model/submission) | Submission the File is a part of | \*required ## File role options Status options for grant | Value | Description | | ------------ | ------------------------------------------------------------------------------------------------- | | manuscript | Author accepted manuscript | | supplemental | Supplemental material for the [Publication](/developer-documentation/pass-core/model/publication) | | figure | An image, data plot, map, or schematic | | table | Tabular data | # Funder Funder / sponsor of a [Grant](/developer-documentation/pass-core/model/grant). | Field | Type | Description | | -------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | id\* | String | Autogenerated identifier of object | | name\* | String | Funder name | | url | String | Funder URL | | localKey | String | Local key assigned to the funder within the researcher's institution to support matching between PASS and a local system. The value is in the form of : `domain:type:value`. For a funder at JHU, an example would be`"johnshopkins.edu:funder:8675309"` | | Relationship | Type | Target | Description | | ------------ | ------ | --------------------------------------------------------- | --------------------------------- | | policy | To One | [Policy](/developer-documentation/pass-core/model/policy) | Policy associated with the Funder | \*required # Grant Grants are imported from the institutional Grant system (FIBI in the case of JHU). They are associated with the Grant's PIs via their [User](/developer-documentation/pass-core/model/user) records. Users of PASS can assign the Grants associated with them to [Submissions](/developer-documentation/pass-core/model/submission). | Field | Type | Description | | ------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | id\* | String | Autogenerated identifier of object | | awardNumber\* | String | Award number from funder | | awardStatus | String | Status of award ([*see list below*](#status-options)) | | localKey | String | A local key assigned to the Grant within the researcher's institution to support matching between PASS and a local system. The value is in the form of : `domain:type:value`. For a grant at JHU, an example would be`"johnshopkins.edu:grant:8675309"` | | projectName\* | String | Title of the research project | | awardDate\* | String | DateTime the grant was awarded | | startDate | String | DateTime the grant started | | endDate | String | DateTime the grant ended | | Relationship | Type | Target | Description | | --------------- | ------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | primaryFunder\* | To One | [Funder](/developer-documentation/pass-core/model/funder) | Funder that is the original source of the funds. This will often be the same as directFunder. | | directFunder\* | To One | [Funder](/developer-documentation/pass-core/model/funder) | Funding organization from which funds are directly received. This will often be the same as primaryFunder. | | pi\* | To One | [User](/developer-documentation/pass-core/model/user) | User who is the Principal investigator | | coPis | To Many | \[[User](/developer-documentation/pass-core/model/user)] | Users who are the co-principal investigators | \*required ## Status options Status options for grant | Value | Description | | ---------- | ------------------------ | | active | Grant currently active | | pre\_award | Award not yet received | | terminated | Grant period is complete | # Journal A Journal is associated with a [Publication](/developer-documentation/pass-core/model/publication). In some cases, the Journal may be important for determining whether the [User](/developer-documentation/pass-core/model/user) needs to manually create a [Submission](/developer-documentation/pass-core/model/submission) through PASS or whether the publisher has a pre-existing arrangement with the target [Repository](/developer-documentation/pass-core/model/repository). Specifically, in the case of the National Institutes of Health Public Access Policy, many Journals already make arrangements to submit the author's accepted manuscript to PubMed Central directly. | Field | Type | Description | | ---------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | id\* | String | Autogenerated identifier of object | | journalName\* | String | Name of the journal | | issns | String\[] | Journal ISSNs - Elements are of the form type:value, where type is one of Online or Print, and value is the usual ISSN value xxxx-xxxx. Examples would be Online:1234-5678 and Print:9876-5432. When a type for an ISSN is not known, it should be stored with an empty type (for example :2468-1357). | | nlmta | String | National Library of Medicine Title Abbreviation | | pmcParticipation | String | This field indicates whether a journal participates in the NIH Public Access Program by sending final published article to PMC. If so, whether it requires additional processing fee ([see list below](#pmc-participation-options)) | \*required ## PMC Participation options These are the possible submission methods relating to PMC deposits. A full description of these methods can be found on the [NIH public access website](https://publicaccess.nih.gov/submit_process.htm) | Value | Description | | ----- | ------------------------------------------------------------------------------------------------------------------------------ | | A | PMC deposit route A. Journals automatically post the paper to PMC | | B | PMC deposit route B. Authors must make special arrangements for some journals and publishers to post the paper directly to PMC | | C | PMC deposit route C. Authors or their designee must submit manuscripts to NIHMS | | D | PMC deposit route D. Some publishers will submit manuscripts to NIHMS | # Policy A Policy describes the access compliance requirements for a specific [Funder](/developer-documentation/pass-core/model/funder) or Institution. These Policies are used to determine which [Repositories](/developer-documentation/pass-core/model/repository) a [Publication](/developer-documentation/pass-core/model/publication) should be [submitted](/developer-documentation/pass-core/model/submission) to in order to be in compliance with its associated [Grants](/developer-documentation/pass-core/model/grant) and the User's institutional policies. | Field | Type | Description | | ----------- | ------ | ------------------------------------------------------------- | | id\* | String | Autogenerated identifier of object | | title\* | String | Title of policy e.g. "NIH Public Access Policy" | | description | String | Several sentence description of policy | | institution | String | URI identifying the Institution whose Policy this is (unused) | | policyUrl | String | URL to the actual policy on the policy-owner's page | | Relationship | Type | Target | Description | | -------------- | ------- | ------------------------------------------------------------------- | ------------------------------------- | | repositories\* | To Many | [Repositories](/developer-documentation/pass-core/model/repository) | Repositories that satisfy this policy | \*required # Publication Publication metadata to be associated with one or more [Submission](/developer-documentation/pass-core/model/submission) | Field | Type | Description | | ------------------- | ------ | --------------------------------------------------------------------- | | id\* | String | Autogenerated identifier of object | | title\* | String | Title of work represented by Submission e.g. the title of the article | | publicationAbstract | String | Abstract for work represented by Submission | | doi | String | DOI of item being submitted, if available | | pmid | String | PubMed unique identifier (PMID) of item being submitted, if available | | volume | String | Volume of journal that contains item (if article) | | issue | String | Issue of journal that contains item (if article) | | Relationship | Type | Target | Description | | ------------ | ------ | ----------------------------------------------------------- | ----------------------------------------------- | | journal | To One | [Journal](/developer-documentation/pass-core/model/journal) | Journal the publication is part of (if article) | \*required # Repository A Repository is the target of a [Deposit](/developer-documentation/pass-core/model/deposit). It is a platform where [copies](/developer-documentation/pass-core/model/repositorycopy) of [publications](/developer-documentation/pass-core/model/publication) can be [deposited](/developer-documentation/pass-core/model/deposit) in order to comply with [Funder](/developer-documentation/pass-core/model/funder) and institutional access [policies](/developer-documentation/pass-core/model/policy). | Field | Type | Description | | --------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | id\* | String | Autogenerated identifier of object | | name\* | String | Name of repository e.g. "PubMed Central" | | description | String | Several sentence description of repository | | url | String | URL to the homepage of the repository so that PASS users can view the platform before deciding whether to participate in it | | agreementText | String | The legal text that a `submitter` must agree to in order to submit a publication to this Repository | | formSchema | String | *(deprecated)* Stringified JSON representing a form template to be loaded by the front-end when this Repository is selected | | integrationType | String | Type of integration that PASS has with the Repository ([*see list below*](#integration-type-options)) | | repositoryKey | String | Key that is unique to this Repository instance within PASS. Used to look up the Repository when its URI is not available e.g., prior to the creation of this Repository resource in Fedora. See below for a [*list of currently used keys*](#repository-key-values). | | schemas | String\[] | Contains an array of relative URIs that the pass-core metadata service can resolve to JSON schema documents describing the repository's metadata requirements | \*required ## Integration type options These are the possible types of integration a Repository can have with PASS. | Value | Description | | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | full | PASS can make [Deposits](/developer-documentation/pass-core/model/deposit) to this Repository, and will received updates about its status | | one-way | PASS can make [Deposits](/developer-documentation/pass-core/model/deposit) to this Repository but will not automatically receive updates about its status | | web-link | A deposit cannot automatically be made to this Repository from PASS, only a web link can be created. | ## Repository key values These are the repository keys currently used in PASS. This list will grow as more repositories are supported. | Value | Repository name | | ------------ | ------------------------------------------------------------------------ | | pmc | [PubMed Central](https://www.ncbi.nlm.nih.gov/pmc/) | | jscholarship | [Johns Hopkins JScholarship](https://jscholarship.library.jhu.edu/) | | eric | [Education Resources Information Center](https://eric.ed.gov/) (ERIC) | | dec | [Development Experience Clearinghouse](https://dec.usaid.gov/dec/) (DEC) | | dash | [Harvard DASH](https://dash.harvard.edu/) | | inveniordm | [Invenio RDM](https://inveniosoftware.org/products/rdm/) | # RepositoryCopy A Repository Copy represents a copy of a [Publication](/developer-documentation/pass-core/model/publication) that exists in a target [Repository](/developer-documentation/pass-core/model/repository). The Repository Copy either (1) was the result of an accepted [Deposit](/developer-documentation/pass-core/model/deposit) from PASS, in which case there would be a link to the Copy from the related Deposit record, or (2) was created outside of PASS by some other process. In the second case, PASS stores information to help determine whether a Publication is already compliant with the repository's requirements. | Field | Type | Description | | ------------ | --------- | --------------------------------------------------------------------------------------------------- | | id\* | String | Autogenerated identifier of object | | externalIds | String\[] | IDs assigned to this entity by the target repository | | copyStatus\* | String | Status of the copy in the external repository's workflow ([*see list below*](#copy-status-options)) | | accessUrl | String | URL to access the item in the repository, could allow Users to see the final result | | Relationship | Type | Target | Description | | ------------- | ------ | ------------------------------------------------------------------- | ---------------------------------- | | publication\* | To One | [Publication](/developer-documentation/pass-core/model/publication) | Publication that this is a copy of | | repository\* | To One | [Repository](/developer-documentation/pass-core/model/repository) | Repository being deposited to | \*required ## Copy status options These are the possible statuses for a Deposit in the order they could occur. Note that not all repositories will go through every status. | Value | Description | | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | accepted | The target [Repository](/developer-documentation/pass-core/model/repository) has indicated that the Deposit has been accepted | | in-progress | The target [Repository](/developer-documentation/pass-core/model/repository) is processing the Deposit | | stalled | The target [Repository](/developer-documentation/pass-core/model/repository) has detected a problem that has caused the progress to stall. This will likely require some direct interaction with the repository to re-initiate the process. Examples include when there are incorrect files or when a user did not respond to a validation request in a reasonable amount of time. | | complete | The target [Repository](/developer-documentation/pass-core/model/repository) has accepted the Deposit, and publication is pending if not already complete | | rejected | The target [Repository](/developer-documentation/pass-core/model/repository) has rejected the Deposit. | # Submission In order to comply with [funder](/developer-documentation/pass-core/model/funder) and institutional access [policies](/developer-documentation/pass-core/model/policy), [Users](/developer-documentation/pass-core/model/user) may be required to submit their [Publications](/developer-documentation/pass-core/model/publication) to one or more [Repositories](/developer-documentation/pass-core/model/repository). A Submission is associated with one `submitter` and one Publication. It encapsulates a User satisfying one or more Policies relevant to their Publication by either (1) [Deposits](/developer-documentation/pass-core/model/deposit) initiated in PASS or (2) [Copies](/developer-documentation/pass-core/model/repositorycopy) of the publication that already exist in the target repositories. The User can start a Submission by describing their Publication and attaching relevant [Grants](/developer-documentation/pass-core/model/grant) to it. The PASS system will use this information to determine which policies apply, and will help the User send the Publication out to the Repositories that will fulfill them. Note that the source of a Submission record is not always a PASS User. In some instance, Submissions are created as a result of an import process designed to ensure that the User can see data relevant to their compliance with Policies in a uniform way. | Field | Type | Description | | ----------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | id | String | Autogenerated identifier of object | | metadata | String | Stringified JSON representation of metadata captured by the relevant repository forms. This will hold extended metadata relevant to the repositories selected in the Submission workflow. It may include fields such as embargoEndDate, embargoText, pmid, and anything else required by specific repositories etc. | | source\* | String | Indicates whether the record came from outside of PASS as an import, or was created through the system ([*see list below*](#source-options)) | | submitted\* | boolean | When true, this value signals that the Submission will no longer be edited by the User. It indicates to Deposit services that it can generate Deposits for any Repositories that need one. This becomes "true" when the User clicks "submit" in the UI. For Submissions generated by loader processes this value will be false when the User must complete the Submission, or true if it is merely pointing to an existing copy of the Publication in a Repository. | | submittedDate | String | DateTime the record was submitted by the [User](/developer-documentation/pass-core/model/user) through PASS | | submissionStatus\* | String | The current status of the Submission, derived from the [Deposit](/developer-documentation/pass-core/model/deposit) status(es), [RepositoryCopy](/developer-documentation/pass-core/model/repositorycopy) status(es) and the `eventType` of the most recent [SubmissionEvent](/developer-documentation/pass-core/model/submissionevent) ([*see list below*](#submission-status-options)) | | aggregatedDepositStatus | String | Current combined status of Deposits, utilized by Deposit Services. The initial status of a new Submission will be "not-started" ([*see list below*](#aggregated-deposit-status-options)) | | submitterName | String | Name of submitter. This field is used when a preparer nominates a submitter that is not yet a PASS [User](/developer-documentation/pass-core/model/user). The name is temporarily stored for use in communications with the submitter until a `User.id` is available. Once there is a URI for `submitter`, the `submitterName` should be null. | | submitterEmail | String | Email of submitter, formatted as a URI e.g. `mailto:first.last@example.com`. This field is used when a preparer nominates a submitter that is not yet a PASS [User](/developer-documentation/pass-core/model/user). The email value is temporarily stored for use in communications with the submitter until a `User.id` is available. Once there is a URI for `submitter`, the `submitterEmail` should be null. | | Relationship | Type | Target | Description | | ----------------- | ------- | ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | publication\* | To One | [Publication](/developer-documentation/pass-core/model/publication) | Publication represented in this Submission | | repositories\* | To Many | [Repository](/developer-documentation/pass-core/model/repository) | Repositories that this Publication will exist in when the Submission is completed | | submitter | To One | [User](/developer-documentation/pass-core/model/user) | User responsible for submitting the Submission. The User will be the individual who either (a) created this Submission through PASS, thus claiming responsibility; (b) was designated as `submitter` by a `preparer`; or, (c) has been assigned the role based on their being PI of an associated [Grant](/developer-documentation/pass-core/model/grant). When this value is null, it indicates there is not yet a User record for the designated submitter. In this instance there should be a value in `submitterName` and `submitterEmail`. | | preparers | To Many | [User](/developer-documentation/pass-core/model/user) | Users who prepared, or who could contribute to the preparation of, the Submission. Preparers can edit the content of the Submission (describe the [Publication](/developer-documentation/pass-core/model/publication), add [Grants](/developer-documentation/pass-core/model/grant), select [Repositories](/developer-documentation/pass-core/model/repository)) but cannot approve Repository agreements, or submit the publication - these tasks must be performed by the `submitter`. | | grants | To Many | [Grant](/developer-documentation/pass-core/model/grant) | Grants that are associated with the User and are relevant to the Publication being submitted | | effectivePolicies | To Many | [Policy](/developer-documentation/pass-core/model/policy) | Policies that will be satisfied via deposit through PASS | \*required ## Submission status options Below are the possible values for the `Submission.submissionStatus` field. They are listed in the order they would typically occur, and with an indication of the arrangement of the data that will result in this status. Note that not all Submissions will go through every status. | Value | Description | Data determining status | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | draft | Newly created Submissions *by the UI* will have this status by default. Only unsubmitted Submissions should have this status. Submissions created by other processes may use a different status. | Default status for Submissions newly created by a user interacting with the UI. | | manuscript-required | When the PASS system identifies a need for a User to submit a Publication to a particular Repository, it will create a new Submission record with this status in order to prompt the User to provide the document and complete the Submission. For example, PASS imports information from the NIH Public Access Compliance system, which contains information about out of compliance publications - these will appear in the PASS system for PI of the corresponding Grant with the label `manuscript-required`. | New Submissions of this type are created with this status already set | | approval-requested | A Submission was prepared by a `preparer` but now needs the `submitter` to approve and submit it or provide feedback. | `Submission.submitted=false` and the most recent [SubmissionEvent](/developer-documentation/pass-core/model/submissionevent) has `eventType=approval-requested`. | | changes-requested | A Submission was prepared by a `preparer`, but on review by the `submitter`, a change was requested. The Submission has been handed back to the `preparer` for editing. | `Submission.submitted=false` and the most recent [SubmissionEvent](/developer-documentation/pass-core/model/submissionevent) has `eventType=changes-requested`. | | cancelled | A Submission was prepared and then cancelled by the `submitter` or `preparer` without being submitted. No further edits can be made to the Submission. | `Submission.submitted=false` and the most recent [SubmissionEvent](/developer-documentation/pass-core/model/submissionevent) has `eventType=cancelled`. | | submitted | The submit button has been pressed through the UI. From this status forward, the Submission becomes read-only to both the `submitter` and `preparers`. This status indicates that either (a) the Submission is still being processed, or (b) PASS has finished the Deposit process, but there is not yet confirmation from the Repository that indicates the Submission was valid. Some Submissions may remain in a `submitted` state indefinitely depending on PASS's capacity to verify completion of the process in the target [Repository](/developer-documentation/pass-core/model/repository). | `Submission.submitted=true`, and the Publication associated with the Submission is in a positive status for each Repository i.e. it's `RepositoryCopy.copyStatus` is not `rejected` or `stalled`, and in the absence of a RepositoryCopy, the `Deposit.depositStatus` is not `rejected`. | | needs-attention | Indicates that a [User](/developer-documentation/pass-core/model/user) action may be required outside of PASS. The Submission is stalled or has been rejected by one or more [Repository](/developer-documentation/pass-core/model/repository) | The `copyStatus` of one or more [RepositoryCopy](/developer-documentation/pass-core/model/repositorycopy) for the Submission is `rejected` or `stalled`. In the absence of a `RepositoryCopy`, the [Deposit](/developer-documentation/pass-core/model/deposit) for that Repository has a `depositStatus` of `rejected`. To be clear, a positive status on the RepositoryCopy can override a negative status on the Deposit. | | complete | The target repositories have all received a copy of the Submission, and have indicated that the Submission was successful. | There is a RepositoryCopy with `repoCopyStatus=complete` for each of the target repositories. | ## Aggregated Deposit status options These are the possible statuses for a Submission's aggregatedDepositStatus field. They are listed in the order they would occur. Note that not all Submissions will go through every status. | Value | Description | | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | not-started | No [Deposits](/developer-documentation/pass-core/model/deposit) have been initiated for the Submission | | in-progress | One or more [Deposits](/developer-documentation/pass-core/model/deposit) for the Submission have been initiated, and at least one has not reached a status of "accepted" or "rejected" | | failed | One or more [Deposits](/developer-documentation/pass-core/model/deposit) for the Submission has a status of "failed" | | accepted | All related [Deposits](/developer-documentation/pass-core/model/deposit) for the Submission have a status of "accepted" | | rejected | One or more [Deposits](/developer-documentation/pass-core/model/deposit) for the Submission has a status of "rejected" | ## Source options These are the possible sources of a Submission | Value | Description | | ----- | ----------------------------------------------------------------------------------------------------------- | | pass | Submission record was created or submitted via the PASS user interface | | other | Submission record was automatically created by harvesting and ingesting from a 3rd party service e.g. NIHMS | # SubmissionEvent The SubmissionEvent model captures significant events that are performed by an agent and occur against a [Submission](/developer-documentation/pass-core/model/submission). Currently, the agent is a PASS [User](/developer-documentation/pass-core/model/user). The definition of "significant" will evolve depending on which events are useful to capture in order to trigger notifications, or form an audit trail. The events that are currently deemed significant for capture are documented under [`eventType`](#event-type-options). | Field | Type | Description | | --------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | id\* | String | Autogenerated identifier of object | | eventType\* | String | The type of event ([*see list below*](#event-type-options)) | | performedDate\* | String | DateTime the event was performed by the [User](/developer-documentation/pass-core/model/user) | | performerRole | String | Role of the person performing the event ([*see list below*](#performer-role-options)) | | comment | String | A comment relevant to the SubmissionEvent. For example, when a `changes-requested` event occurs, the User might add a comment through the UI to specify what changes are needed. | | link | String | A URI for a resource relevant to the SubmissionEvent. For example, when a `changes-requested` event occurs, this may contain an Ember application URL to the affected Submission. | | Relationship | Type | Target | Description | | ------------- | ------ | ----------------------------------------------------------------- | ----------------------------------------- | | performedBy\* | To One | [User](/developer-documentation/pass-core/model/user) | User responsible for performing the event | | submission\* | To One | [Submission](/developer-documentation/pass-core/model/submission) | Submission that the event relates to | \*required ## Event type options The following describes the types of events that might be recorded as SubmissionEvents. | Value | Description | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | approval-requested-newuser | A Submission was prepared by a `preparer` on behalf of a person who does not yet have a [User](/developer-documentation/pass-core/model/user) record in PASS. The `preparer` is requesting that the `submitter` join PASS and then approve and submit it or provide feedback. | | approval-requested | A Submission was prepared by a `preparer` who is now requesting that the `submitter` approve and submit it or provide feedback. | | changes-requested | A Submission was prepared by a `preparer`, but on review by the `submitter`, a change was requested. The Submission has been handed back to the `preparer` for editing. | | cancelled | A Submission was prepared and then cancelled by the `submitter` or `preparer` without being submitted. No further edits can be made to the Submission. | | submitted | The submit button has been pressed through the UI. | ## Performer role options The following describe the roles of people who might perform a SubmissionEvent. | Value | Description | | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | preparer | An individual who can prepare a Submission on behalf of another User - select the Publication, Repositories, Files, and Grants - but cannot approve the Repository agreements or submit the record for Deposit. | | submitter | An individual responsible for a Submission. A person with this role can do all of the tasks that a `preparer` can do, but also approve any Repository agreements and submit the record for Deposit. | # User A User of the PASS system. This includes preferred person information that can be used to autopopulate [Contributor](https://github.com/eclipse-pass/pass-documentation/blob/main/developer-documentation/pass-core/model/Contributor.md) records. | Field | Type | Description | | ------------ | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | id\* | String | Autogenerated identifier of object | | username\* | String | Unique login name used by User | | firstName | String | First name(s) of User | | middleName | String | Middle name(s) of User | | lastName | String | Last name(s) of User | | displayName | String | Name for display. Separate names may not be available, but a person should always at least have a display name. | | email | String | Contact email for User | | affiliation | String\[] | The affiliation(s) of the User with their institution, for example `STAFF@inst.edu`. An institution may have multiple organizational units, and a User may have a different affiliation with any given OU. A User having an affiliation with multiple OUs in an institution would have multiple values, for example `FACULTY@medicine.inst.edu` and `STUDENT@engineering.inst.edu`. | | locatorIds\* | String\[ ] | A list of ids associated with the user by various system that PASS interacts with. The value of each entry would be in the form of : `domain:type:value`. For example, `["johnshopkins.edu:hopkinsid:DRA2D", "johnshopkins.edu:employeeid:12345", "johnshopkins.edu:jhed:bostaur1"]`. The following values for `type` are considered deprecated: `jhed`, `hopkinsid`. The preferred types are `eppn` and `unique-id`, respectively. | | orcidId | String | ORCID ID for the User | | roles\* | String\[] | User roles ([*see list below*](#role-options)) | \*required ## Role options Role options for User. | Value | Description | | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | submitter | User who can view and manage Submissions for personal Publications or those associated with their own Grants | | admin | User who can manage Submissions for personal Publications or those associated with their own Grants, as well as view Submissions for all Grants | # PASS UI PASS is an [Ember.js](https://emberjs.com/) application which provides a unified user interface that allow its users to deposit manuscripts into multiple repositories as required by applicable funding agency's public access policies PASS communicates with an [Elide-based API](https://github.com/yahoo/elide), and [pass-core](https://github.com/eclipse-pass/pass-core) on the backend that serves json in conformance with the [JSON:API spec](https://jsonapi.org/). ## Technologies Utilized * [Git](https://git-scm.com/) * [Docker](https://www.docker.com/) * [Docker Compose](https://docs.docker.com/compose/) * [Node.js](https://nodejs.org/) >= 24 * [pnpm](https://pnpm.io/) 9 ## Installation * Clone the repository using the following command: `git clone https://github.com/eclipse-pass/pass-ui` * Change into the directory that was cloned: `cd pass-ui` * Run the following command: `pnpm i` ## Running `pass-ui` Inside the Docker Environment The default environment for running `pass-ui` locally utilizes [pass-docker](https://github.com/eclipse-pass/pass-docker). `pass-docker` can run with `pass-ui` running inside of the docker network in its own service and docker container with these [instructions](https://github.com/eclipse-pass/pass-documentation/tree/development/developer-documentation/pass-docker/README.md). This environment is not as conducive for active development of `pass-ui`, depending on your host machine's operating system you might be able to run `pass-ui` on your host machine outside of the docker network and use these [instructions](https://github.com/eclipse-pass/pass-documentation/tree/development/developer-documentation/running-pass-ui-on-your-host-machine.md) to forward traffic from the docker network to your host machine. ### Building for the Docker Environment GitHub automations are in place to produce production builds during a release. If you want to build a Docker image for local testing: ```sh pnpm build pnpm build:docker ``` This will perform a Vite production build to `dist/` and create a new pass-ui Docker image tagged with the version from `package.json`. ## Running `pass-ui` Outside of the Docker Environment It can be a better development experience to run `pass-ui` outside of `pass-docker`. To do this complete the following steps: ### Configure the Docker Environment You will need to configure `pass-core` to load the UI from `localhost:4200`. This can be done by seting the environment variable `PASS_CORE_APP_LOCATION` to `http://host.docker.internal:4200/app/` in `.env`. This will bypass the `pass-ui` container. Then simply use docker compose like normal. ``` docker compose -f docker-compose.yml -f eclipse-pass.local.yml ``` Note: You may need to investigate other ways of accessing the host machine network, [see](https://docs.docker.com/desktop/networking/#i-want-to-connect-from-a-container-to-a-service-on-the-host). You may also consider stopping the pass-ui container. ### Run pass-ui on a Host Machine Start the Vite development server on port 4200: ``` pnpm start ``` The development server includes API proxy configuration for `/data`, `/user`, `/schema`, `/doi`, `/policy`, and `/file` routes, forwarding them to `localhost:8080` (pass-core). To run a production build locally with the Vite preview server: ``` pnpm start:production ``` To run in development mode with pass-core, you may need to set a permissive CSP in pass-docker. Edit `.env` to set `PASS_CORE_APP_CSP` to `"default-src * 'unsafe-inline' 'unsafe-eval' data: blob:;"`. ## Test Users Refer to the LDAP service in `pass-docker` for [a list](https://github.com/eclipse-pass/pass-docker/blob/main/ldap/pass.ldif) of test users. Each has a password of `moo`. ## Configuration The configuration for the docker environment occurs in the `pass-docker` [.env](https://github.com/eclipse-pass/pass-docker/blob/main/.env). A list of environment variables related to `pass-ui`, and other services, can be found there or in other [override .env files](https://github.com/eclipse-pass/pass-docker/blob/main/.eclipse-pass.local_env). The application also gets "branding" configuration from a `config.json` file, with a default implementation found in the `public/` directory, which is automatically made available by default at `/app/config.json`. `config.json` ```js { "branding": { "homepage": "https://www.eclipse.org/org/foundation/", "logo": "ef/eclipse_foundation_logo_wo/EF_WHT-OR_png.png", "favicon": "favicon.ico", "stylesheet": "/app/branding.css", "overrides": "/app/branding-overrides.css", "pages": { "showPagesNavBar": false }, "error": { "icon": "/app/error-icon.png" } } } ``` The base theme styles can be found in `branding.css`. There are default fallback styles which can be overridden to customize the appearance of the UI. It is recommended to override these styles through a `branding-overrides.css` file. An example of these overrides to the base styles can be found [here](https://github.com/eclipse-pass/pass-ui/blob/main/public/branding-overrides.css). ## Testing To run the unit/integration/acceptance tests: ``` pnpm test:ember ``` To run a single test or module: ``` pnpm test:ember --filter="module name or test name" ``` To run linting and tests together: ``` pnpm test ``` ## Linting This project uses `eslint`, `ember-template-lint` and `prettier` to enforce style decisions and code formatting. Consider installing [an integration tool](https://prettier.io/docs/en/editors.html). This project uses [husky](https://github.com/typicode/husky) to run a command from [lint-staged](https://github.com/okonet/lint-staged) to run `eslint --fix` and `prettier --write` over the staged files in a pre-commit hook. If issues arise during a commit it might be because either one or both of these commands has failed. Check the output in the terminal for what failures have occurred. There are also scripts defined in the [package.json](https://github.com/eclipse-pass/pass-ui/blob/main/package.json) that can manually lint check the project. ## CI Testing and Linting utilizes the [ci.yml workflow](https://github.com/eclipse-pass/pass-ui/blob/main/.github/workflows/ci.yml) in Github when a pull request is opened and when there is a push to the `main` branch. More information regarding CI/CD can be found at the following link: [PASS Continuous Integration and Continuous Delivery](/infrastructure-documenation/ci-cd) ## Related Documentation: * [ember.js](https://emberjs.com/) * [Vite](https://vite.dev/) * [WarpDrive](https://warp-drive.io/) * Development Browser Extensions * [ember inspector for chrome](https://chrome.google.com/webstore/detail/ember-inspector/bmdblncegkenkacieihfhpjfppoconhi) * [ember inspector for firefox](https://addons.mozilla.org/en-US/firefox/addon/ember-inspector/) * [PASS Continuous Integration and Continuous Delivery](/infrastructure-documenation/ci-cd) # Data Loaders ## PASS Data Loaders The PASS Data Loaders comprise three components in the [Pass Support](https://github.com/eclipse-pass/pass-support) repository: [Pass Journal Loader](https://github.com/eclipse-pass/pass-support), [Pass NIHMS Loader](https://github.com/eclipse-pass/pass-support), [Pass Grant Loader](https://github.com/eclipse-pass/pass-support). These three components are responsible for loading data from external sources into PASS. ## Summary All three loaders are Java JAR command line applications and can be run from any platform that can run Java applications. These applications utilize system properties and can be configured to run with different parameters. ### [Grant Loader](/developer-documentation/data-loaders/grant-loader) The Grant Loader is designed to automate the ingestion and processing of grant data from various sources into PASS. It handles the loading of grants using predefined configurations and mappings, ensuring the correct representation and association of grant data. Since there are predefined fields that are required to represent a grant and institutions may have varying representations, specific implementations may be needed in order to accommodate other institutions. The design of the Grant Loader is flexible in that it can accommodate development of connectors to varying data sources. ### [Journal Loader](/developer-documentation/data-loaders/journal-loader) The Journal Loader facilitates the automated loading of journal data, particularly from sources such as [PubMed](https://pubmed.ncbi.nlm.nih.gov/) and [PubMed Central](https://www.ncbi.nlm.nih.gov/pmc/). The Journal Loader is responsible for streamlining the process of updating and maintaining journal entries in PASS. ### [NIHMS Loader](/developer-documentation/data-loaders/nihms-loader) The NIHMS Loader specifically targets the loading and transformation of manuscript submission data from the NLM’s Public Access Compliance Monitor (PACM) into PASS. This enables publications in PASS to be updated appropriately with their publication information that is in PubMed Central. The PACM [user guide](https://www.ncbi.nlm.nih.gov/pmc/utils/pacm/static/pacm-user-guide.pdf) explains the background of the PACM system and its data. It will assist in setting up the appropriate accounts in order to access the PACM system and API. ## Technologies Utilized * [Docker](https://www.docker.com/products/docker-desktop/) for running and testing the applications. * [Java 17+](https://www.oracle.com/java/technologies/downloads/) for the application development. * [Spring Boot](https://spring.io/projects/spring-boot) for the application framework. ## Related Information The following resources cover external resources for data extraction and the infrastructure used to run the Data Loaders: * PubMed Central Resources * [PACM Guide](https://www.ncbi.nlm.nih.gov/pmc/utils/pacm/static/pacm-user-guide.pdf) * [PMC APIs](https://www.ncbi.nlm.nih.gov/pmc/tools/developers/#pmc-apis) * NIHMS Resources * [NIH Manuscript Submission](https://www.nihms.nih.gov) * [NIH Manuscript Submission Process](https://www.nihms.nih.gov/about/overview/) * PASS * [PASS Architecture](/welcome-guide/deployment-architecture) * [PASS Support](https://github.com/eclipse-pass/pass-support) # Grant Loader The Grant Loader ingests grant data from an institution and maps the data to the appropriate PASS Objects in the PASS Data Model. ## Summary This module comprises code for retrieving grant data from some kind of data source, and using that data to update the PASS backend. Typically, the data pull will populate a data structure, which will then be consumed by a loader class. While this sounds simple in theory, there are several considerations which may add to the complexity of implementations. An implementor must first determine the data requirements for PASS and then map these requirements to data available from the data source. Additional data from other services may be needed to populate the data structures to be loaded into PASS. For data loading, the implementor may need to support different modes of ingesting data. Additional logic may be needed in the data loading process to resolve the fields in the data assembled by the pull process. For example, several systems may be updating PASS objects, and that other services may be more authoritative for certain fields than the service providing the grant data. The JHU implementation is complex regarding these issues. ## Knowledge Needed / Skills Inventory * Development of the Grant Loader * Programming in Java * Basic understanding of your institution's grant data * Running the Grant Loader * CLI commands ## Technologies Utilized * [Docker](https://www.docker.com/products/docker-desktop/) * [Java 17+](https://www.oracle.com/java/technologies/downloads/) * [Spring Boot](https://spring.io/projects/spring-boot) ## Technical Deep Dive ### Configuring using Spring Boot Profiles The grant loader uses Spring Boot Profiles to select the appropriate classes to be used for a given institution. There is a property in application.properties named `spring.profiles.active` that needs to be set when starting the grant loader. This property can be set at runtime as well using the normal spring boot configuration functionality: [Spring Boot Configuration](https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.external-config). The code has been factored to ease development for multiple institutions. These are the institution-specific classes which typically need to be implemented and annotated with the `@Profile` annotation: #### Connector The Connector class connects to the data store for an institution's implementation, and operates on the data to supply, in as standard a form as possible, the data to be consumed by the Updater. #### Updater The Updater class takes the data supplied by the Connector and creates or updates the corresponding objects in the PASS repository accordingly. There is a Default class whose children may override certain substantive methods if the local policies require. ### Profiles #### JHU The JHU implementation is used to pull data from the COEUS/FIBI database views for the purpose of performing regular updates. We identify grants which have been updated since a specific time (typically the time of the previous update), join this with user and funder information associated with the grant, and then use this information to update the data in the PASS backend. The JHU implementation also treats the COEUS/FIBI database as authoritative for all fields in the data. If a grant is being passed in for update, it is assumed that all records for that grant are included in the input. ### Grant Data In order for PASS to map grant data to the associated Objects within PASS, the Grant Loader needs to ingest a CSV file with the following fields and data types: | Column | Type/Size | Required | Description | | --------------------- | --------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | GRANT\_NUMBER | TEXT/255 | Y | Unique identifier for the grant (institutional Grant ID) | | GRANT\_TITLE | TEXT/255 | Y | The title of the grant | | AWARD\_NUMBER | TEXT/255 | N | Unique identifier for the award | | AWARD\_STATUS | TEXT/255 | N | The status of the award. Valid values: active, pre-award, terminated | | AWARD\_DATE | DATE | N | The date the grant award was created. Format: YYYY-MM-DD or YYYY-MM-DD HH:MM:SS.SSS if time is required. Date/Time is (UTC timezone) | | AWARD\_START | TIMESTAMP | Y | The timestamp the grant award begins. Format: YYYY-MM-DD or YYYY-MM-DD HH:MM:SS.SSS if time is required. Date/Time is (UTC timezone) | | AWARD\_END | TIMESTAMP | Y | The timestamp the grant award ends. Format: YYYY-MM-DD or YYYY-MM-DD HH:MM:SS.SSS if time is required. Date/Time is (UTC timezone) | | PRIMARY\_FUNDER\_NAME | TEXT/255 | N | The Primary Funder Name (Funder of original source of funds). If not set, the Direct Funder will also be set as Primary Funder. | | PRIMARY\_FUNDER\_CODE | TEXT/255 | N | The Primary Funder unique identifier (institutional Funder ID). If not set, the Direct Funder will also be set as Primary Funder. | | DIRECT\_FUNDER\_NAME | TEXT/255 | Y | The Direct Funder Name (Funder from which funds are directly received) | | DIRECT\_FUNDER\_CODE | TEXT/255 | Y | The Direct Funder unique identifier (institutional Funder ID) | | PI\_FIRST\_NAME | TEXT/255 | Y | First name of PI | | PI\_MIDDLE\_NAME | TEXT/255 | N | Middle name of PI | | PI\_LAST\_NAME | TEXT/255 | Y | Last name of PI | | PI\_EMAIL | TEXT/255 | Y | Email address of PI | | PI\_INSTITUTIONAL\_ID | TEXT/128 | Y | Institutional User ID of PI. This is typically the User ID in the institution's Identity Access Management system. This value is optional if PI\_EMPLOYEE\_ID exists. If User ID exists in this record, it is important that the User ID is available to PASS during the authentication process. | | PI\_EMPLOYEE\_ID | TEXT/128 | Y | Employee ID of PI. This value is optional if PI\_INSTITUTIONAL\_ID exists. If Employee ID exists in this record, it is important that the Employee ID is available to PASS during the authentication process. | | PI\_ROLE | TEXT/1 | Y | Role of PI on grant (PI or Co-PI). Valid values: P, C. P=PI, C=Co-PI | | UPDATE\_TIMESTAMP | TIMESTAMP | N | Last update timestamp. Format: YYYY-MM-DD or YYYY-MM-DD HH:MM:SS.SSS if time is required. Date/Time is (UTC timezone) | ### Usage Refer to the application.properties file to determine which properties that need runtime values set. The grant loader is a spring boot application, so use the standard Spring Boot configuration functionality according to the [Spring Boot Configuration](https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.external-config) documentation. Here is an example using Java system properties `-D`: ```shell java -jar jhu-grant-loader-exec.jar -a load file:./grant-data.csv ``` #### Arguments Run the above command with `-h` to display a full list of arguments for the grant loader. In this example below using the `-a` parameter instructs the grant loader to load data from a file CSV file. The `-a` parameter action can be set to either `pull` or `load`. Use `pull` to extract data from the grant source system and store it in a file, or `load` to ingest data from an existing file into PASS. ```shell java -jar jhu-grant-loader-exec.jar -a load file:./grant-data.csv ``` In another example below, `startDateTime` and `awardEndDate` are used as parameters to limit the date range of the grant data. Since no action is specified, the default is to perform a pull followed directly by a load, using the default connection source. ```shell java -jar jhu-grant-loader-exec.jar -startDateTime -awardEndDate ``` ### Running the Grant Loader in Docker #### Run PASS Docker Since the Grant Loader will load data from a CSV into PASS, you will need an instance of PASS running. The quickest way to accomplish this is to run [PASS docker](/welcome-guide/setup-run-pass). Start `pass-docker` in local mode by running with the `docker-compose.yml` and `eclipse.pass.local.yml` configurations: ```shell docker compose -f docker-compose.yml -f eclipse-pass.local.yml up -d --no-build --quiet-pull ``` Once pass-docker is up and the loader container is done running, open a browser and go to and login with nih-user. More details about this account can be found on the [PASS docker](/welcome-guide/setup-run-pass) page. Go to Grants tab to view all the grants. This page will be empty, but after running the Grant Loader it will have all the grants from the CSV file provided. #### Setup Grant Loader Test Directory 1. Create a directory named `grantloadertest`. 2. Change to the directory: `cd grantloadertest`. 3. Create the following files: * `grant_update_timestamps` (empty) * `policy.properties` (empty) * `env.list` with content: ``` APP_HOME_ENV=/data/grantloader POLICY_PROP_PATH=file:/data/grantloader/policy.properties PASS_CORE_URL=http://localhost:8080 PASS_CORE_USER= PASS_CORE_PASSWORD= ``` 4. Copy your grant CSV file to the `grantloadertest` directory. 5. Open a new terminal and cd to the pass-docker directory. #### Running Grant Loader Load For testing purposes, we need to associate nih-user to a grant row in the CSV. 1. Modify one of your grant rows to change the user fields to: ``` Ser,,Nihu,nihuser@jhu.edu,NIHUSER,118110 ``` 2. Open a new terminal window and navigate to `grantloadertest` 3. Run ``` docker run -it -v ./grantloadertest:/data/grantloader --env-file ./grantloadertest/env.list --network host ghcr.io/eclipse-pass/jhu-grant-loader:1.8.0-SNAPSHOT -a load /data/grantloader/.csv` ``` Note: Replace `1.8.0-SNAPSHOT` with the version of the grant loader you want to use. 4. Once done, refresh the Grants tab in the browser to see your grant loaded. Troubleshooting: * If the grant csv contains new Funders, you should figure out PASS policy ID and put the funder local key to policy ID mapping in policy.properties (i.e. funder\_local\_key=pass\_policy\_id) before running the grant loader docker command. * If running on Windows references to the current directory should use `${PWD}` for Powershell or `%cd%` for Windows Command Line. ### Grant Loader Classes & Data Flow Overview 1. Initialization and Configuration: * The application initializes with `GrantLoaderCLI`, which sets up the `GrantLoaderApp` with configurations from `GrantLoaderConfig`. * Spring Boot profiles are used to load institution-specific configurations. 2. Data Retrieval: * `GrantLoaderApp` uses the `GrantConnector` interface to retrieve data from the data source (e.g., database, CSV file). * The `CoeusConnector` implementation (e.g., for JHU) fetches the data and returns it as a list of `GrantIngestRecord` objects. 3. Data Processing: * The `GrantIngestRecord` objects are built by the `CoeusConnecter` by the `retrieveUpdates` method. * The `CoeusConnecter` implements `GrantConnector`, and is specific the COEUS database at JHU. Another institution should have an implementing class for their institution. * A `LocalKey` is built using utility methods from `GrantDataUtils` and is used by the `AbstractDefaultPassUpdater` 4. Data Ingestion: * The processed grant data is passed to the `JhuPassUpdater`, which extends `AbstractDefaultPassUpdater`. * An institution with specific needs for updating their data should extend the `AbstractDefaultPassUpdater`. The `JhuPassUpdater` is specific to JHU implementation. * The `JhuPassUpdater` updates PASS objects (grants, users, funders) in the PASS repository, and interacts with the PassClient to perform the actual create and update operations. 5. Error Handling: * Exceptions specific to the data retrieval or ingestion are handled by `GrantDataException`. * Errors are logged, and appropriate messages are reported to the CLI user via `PassCliException`. 6. Statistics Tracking: * The `PassUpdateStatistics` class tracks the number of grants, funders, and users created or updated. * Statistics are updated in the `PassUpdater` and can be reset or reported. ## Next Step / Institution Configuration Institutional configuration is going to be highly dependent on where the institutional grant data comes from. At JHU, we have a Postgres database and the data is pulled from the database using [AWS Batch and ECS](/welcome-guide/deployment-architecture#pass-deployment-architecture). There can be multiple ways to set up the infrastructure, but the simplest setup is to have a CSV file exported to a directory where the Grant Loader can ingest the file using the `-a` parameter. # Journal Loader ## Journal Loader The Journal Loader is responsible for pulling data from [PubMed Central](https://www.ncbi.nlm.nih.gov/pmc/) (PMC) and the [MEDLINE database](https://www.nlm.nih.gov/medline/medline_home.html), and making the appropriate updates to PASS. ## Journal Loader Summary The Journal Loader parses the PMC type A journal `.csv` file, and/or the [MEDLINE database](https://www.nlm.nih.gov/medline/medline_home.html) `.txt` file, and syncs with the repository by taking the following actions: * Adds journals if they do not already exist * Updates PMC method A participation if it differs from the corresponding resource in the repository. ## Knowledge Needed / Skills Inventory * Development of the Journal Loader * Programming in Java * Basic understanding of the NLM journal loader * Running the Journal Loader * CLI commands ## Technologies Utilized * [Java 17+](https://www.oracle.com/java/technologies/downloads/) * [Spring Boot](https://spring.io/projects/spring-boot) ## Technical Deep Dive ### Usage Using Java system properties to launch the journal loader. Note: Replace the version number in the jar name with the specific version you are using. ```shell java -Dpmc=https://www.ncbi.nlm.nih.gov/pmc/front-page/NIH_PA_journal_list.csv -Dmedline=https://ftp.ncbi.nih.gov/pubmed/J_Medline.txt -Dpass.core.url=http://localhost:8080 -Dpass.core.user=USER -Dpass.core.password=PASS -jar pass-journal-loader-nih-exec.jar ``` #### Properties or Environment Variables The following may be provided as system properties on the command line `-Dprop-value`. `pass.core.url` The base URL for the pass-core REST API such as `http://localhost:8080` `pass.core.user` The pass-core backend user. `pass.core.password` The pass-core backend user password. `dryRun` Do not add or update resources in the repository, just give statistics of resources that would be added or updated `pmc` URL of the PMC "type A" journal .csv file, for example: `medline` URL of the Medline journal file, for example: `LOG.*` Adjust the logging level of a particular component, e.g. `LOG.org.eclipse.pass=WARN` ### Journal Loader Classes & Data Flow Overview #### Data Flow 1. Initialization: * The `Main` class initializes the application and calls the `BatchJournalFinder` and `LoaderEngine` to start processing. 2. File Processing: * `BatchJournalFinder` processes each file using the appropriate reader (`MedlineReader`, `NihTypeAReader`). * The `load` method in `BatchJournalFinder` initiates the process, collects files to be processed. 3. Data Loading: * Processed journal data is passed to `LoaderEngine` to be loaded into the target system. * If a journal is not found then a new one will be created, otherwise it will update the journal. ## Next Step / Institution Configuration Journal loader is simple to configure. It will run on any system that can run Java applications and does not require external account setup. The two sources of data PMC Type A Journals and MEDLINE, do not require accounts to access the data. Similar to the [NIHMS Loader](/developer-documentation/data-loaders/nihms-loader) and [Grant Loader](/developer-documentation/data-loaders/grant-loader), the Journal Loader is run using [AWS Batch and ECS](/welcome-guide/deployment-architecture#pass-deployment-architecture). ## Related Information The following resources are the sources of the journal data that is loaded into pass: * [PubMed Central](https://www.ncbi.nlm.nih.gov/pmc/) * [NLM MEDLINE](https://www.nlm.nih.gov/medline/medline_overview.html) # NIHMS Loader The NIH Manuscript Submission Loader (NIHMS Loader) contains the components required to download, transform, and load Submission information from NIHMS to PASS. ## Summary The NIHMS Loader is a module contained in [Pass-Support](https://github.com/eclipse-pass/pass-support), and is composed of two Java command line tools. The first uses the NIH API to download the CSV(s) containing compliant, non-compliant, and in-process publication information. The second tool reads those files, transforms the data to the PASS data model, and then loads them to PASS. For background information on the NIH Public Access Compliance Monitor (PACM), see the [user guide](https://www.ncbi.nlm.nih.gov/pmc/utils/pacm/static/pacm-user-guide.pdf). Limited information on the API is provided by the [NLM Technical Bulletin](https://www.nlm.nih.gov/pubs/techbull/mj19/brief/mj19_api_public_access_compliance.html) The NIHMS Loader operates in two stages: * Harvests data from the NLM’s Public Access Compliance Monitor (PACM) website about the compliance status of publications written by researcher PIs * It compares PACM data with `Submission` data in PASS and then adds or updates `Submission`, `Publication`, and `RepositoryCopy`. These two processes are separate Java command line interface (CLI) applications: the NIHMS Data Harvest CLI and the NIHMS Transform and Load CLI. ## Knowledge Needed / Skills Inventory * Development of the NIHMS Loader * Programming in Java * Basic understanding of PMC data * REST/HTTP * Running the NIHMS Loader * CLI commands ## Technologies Utilized * [Docker](https://www.docker.com/products/docker-desktop/) * [Java 17+](https://www.oracle.com/java/technologies/downloads/) * [Spring Boot](https://spring.io/projects/spring-boot) ## Technical Deep Dive ### NIHMS Data Harvest CLI The NIHMS Data Harvest CLI uses the NIH API to download the PACM data. The following are required to run this tool: * Java 17+ * Download the latest [docker image](https://github.com/eclipse-pass/pass-support/pkgs/container/pass-nihms-loader) or download the latest [pass-support release](https://github.com/eclipse-pass/pass-support/releases) and compile the `pass-nihms-loader` module * An account for the NIH PACM website, and obtain an API key. The API key is only valid for 3 months, so it will need to be updated periodically. There are a couple of ways to obtain an account, and the process is institution-specific. ### Data Harvest Configuration There are several ways to configure the Data Harvest CLI. Data Harvest CLI is a Spring Boot Application, so it can be configured using [Spring Boot Configuration](https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.external-config) You will need to set values for the following properties: * `nihmsetl.api.url.param.inst` * `nihmsetl.api.url.param.ipf` * `nihmsetl.api.url.param.api-token` The full set of properties for the NIHMS Harvester is listed below. These properties are set in the `resources/application.properties` file in the `nihms-data-harvest` module. | Property | Default Value | Notes | | -------------------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | nihmsetl.data.dir | /data/nihmsloader/data | Defines a directory where the files will be downloaded. If not specified, the default directory will be created for you. | | nihmsetl.api.host | [www.ncbi.nlm.nih.gov](http://www.ncbi.nlm.nih.gov) | The host name of the API. The default should not change. If the API is moved to another host this will be updated by the PASS team. | | nihmsetl.api.scheme | https | The HTTP scheme of the API. The default should not change. | | nihmsetl.api.path | /pmc/utils/pacm/ | The API URL Path. The default should not change. If the API does change the URL, the default value will be updated by the PASS team. | | nihmsetl.http.read-timeout-ms | 30000 | Allow 30 seconds for a request to be read before timing out. | | nihmsetl.http.connect-timeout-ms | 30000 | Allow 30 seconds for establishing connections before timing out. | | nihmsetl.api.url.param.format | csv | The format for the downloaded file. The default should not change. The NIHMS Transform and Load requires a csv file. | | nihmsetl.api.url.param.inst | | Name of the institution making the API request. The value for your organization can be found in the [PACM website](https://www.ncbi.nlm.nih.gov/pmc/utils/pacm). | | nihmsetl.api.url.param.ipf | | IPF (Institutional Profile File) number, the unique ID assigned to a grantee organization in the eRA system. | | nihmsetl.api.url.param.api-token | | The API token retrieved from the [PACM website](https://www.ncbi.nlm.nih.gov/pmc/utils/pacm). The token expires every three months. | | nihmsetl.api.url.param.pdf | See Notes | Date in MM/YYYY format that the PACM data should start from. Can be set using the `-s` harvester command line option). By default this date will be set to the current month, one year ago | | nihmsetl.api.url.param.pdt | See Notes | Date in MM/YYYY format that the PACM data should end. Leave blank to default to the current month. | ### Running the Data Harvester The simplest way to run the data harvester is to use the java command in a CLI with [Java system property option](https://docs.oracle.com/en/java/javase/17/docs/specs/man/java.html), by using the -D parameter. Note if you're running on Windows you may have double quote the parameters, for example `"-Dnihmsetl.api.url.param.inst=my-inst"`. ```shell java -Dnihmsetl.api.url.param.inst=my-inst -Dnihmsetl.api.url.param.ipf=my-ipf -Dnihmsetl.api.url.param.api-token=my-token -jar nihms-data-harvest-cli-exec.jar ``` Once the Data Harvest CLI has been configured, there are a few additional options you can add when running from the command line. By default, all 3 publication statuses - compliant, non-compliant, and in-process CSVs will be downloaded. To download one or two of them, you can add them individually at the command line: ``` -c, -compliant, --compliant - Download compliant publication CSV. -p, -inprocess, --inprocess - Download in-process publication CSV. -n, -noncompliant, --noncompliant - Download non-compliant publication CSV. ``` You can also specify a start date, by default the PACM system sets the start date to 1 year prior to the date of the download. You can change this by adding a start date parameter. This will return all records published since the date provided. The syntax for this parameter is mm-yyyy . ``` -s, -startDate --startDate ``` So, for example, to download the compliant publications published since December 2012, you would do the following: ```shell java -jar nihms-data-harvest-cli-exec.jar -s 12-2012 -c ``` On running this command, files will be downloaded and renamed with a prefix according to their status ("compliant", " noncompliant", or "inprocess") and a timestamp integer e.g. noncompliant\_nihmspubs\_20180507104323.csv. ### NIHMS Transform and Load The NIHMS Data Transform-Load CLI reads data in from CSVs that were downloaded from the PACM system, converts them to PASS compliant data and loads them into the PASS database. Pre-requisites The following is required to run this tool: * Java 17+ * Download the latest [docker image](https://github.com/eclipse-pass/pass-support/pkgs/container/pass-nihms-loader) or download the latest [pass-support release](https://github.com/eclipse-pass/pass-support/releases) and compile the `pass-nihms-loader` module. The jar to run for this process is `nihms-data-transform-load-exec.jar` found in the `nihms-data-transform-load` module. ### Data Transform-Load Configuration There are several ways to configure the Data Transform-Load CLI. Data Transform-Load CLI is a Spring Boot Application, so it can be configured as described here: Spring Boot Configuration You will need to set values for the following props: nihmsetl.repository.id, pass.client.url, pass.client.user and pass.client.password. There are a number of ways to do this with a Spring Boot app, which are described in the link above. Here is an example using Java system properties -D. > java -Dnihmsetl.repository.id=my-repo-id -Dpass.client.url=my-url -Dpass.client.user=my-user -Dpass.client.password=my-pw -jar nihms-data-transform-load-exec.jar ### Running the Data Transform-Load Once the Data Transform-Load CLI has been configured, there are a few additional options you can add when running from the command line. By default, all 3 publication statuses: `compliant`, `non-compliant`, and `in-process` CSVs will be downloaded. To download one or two of them, you can add them individually at the command line: ``` -c, -compliant, --compliant -p, -inprocess, --inprocess -n, -noncompliant, --noncompliant ``` So, for example, to process non-compliant spreadsheets only: ```shell java -jar nihms-data-transform-load-cli-exec.jar -n ``` When run, each row will be loaded into the application and new Publications, Submissions, and RepositoryCopies will be created in PASS as needed. The application will also update any Deposit.repositoryCopy links where a new one is discovered. Once a CSV file has been processed, it will be renamed with a suffix of ".done" e.g. noncompliant\_nihmspubs\_20180507104323.csv.done. To re-process the file, simply rename it to remove the .done suffix and re-run the application. ### Running the Harvester and Data Transform-Load using Docker To run both the Harvest and Data Transform-Load using Docker you will also need [PASS docker](/welcome-guide/setup-run-pass) running, otherwise it will fail on the Transform-Load step. Once PASS Docker is running, use docker pull to get the Pass NIHMS Loader image, and be sure to replace the image tag `1.8.0-snapshot` with the version that you want to pull and run. ```shell docker pull ghcr.io/eclipse-pass/pass-nihms-loader:1.8.0-snapshot ``` Run the docker image with the following environment variables, using the [docker -e parameter](https://docs.docker.com/reference/cli/docker/container/run/). On Windows you may have double quote the parameters, for example `"-eNIHMS_API_INST=YOUR_INST"`. ```shell docker run -eNIHMS_API_INST=YOUR_INST -eNIHMS_API_IPF=YOUR_IPF -eNIHMS_API_TOKEN=YOUR_API_TOKEN ghcr.io/eclipse-pass/pass-nihms-loader:1.8.0-snapshot ``` ### NIHMS Data Harvester Classes and Relationships #### Data Flow Overview * **Initialization**: * `NihmsHarvesterCLIRunner` starts and processes command-line arguments. * `NihmsHarvesterCLI` configures and initializes `NihmsHarvester` using `NihmsHarvesterConfig`. * **URL Construction**: * `NihmsHarvester` uses `UrlBuilder` and `UrlType` to create URLs needed to access NIHMS data. * **Data Download**: * `NihmsHarvesterDownloader` retrieves data from the constructed URLs. * Downloaded data is passed back to `NihmsHarvester`. * **Data Processing**: * `NihmsHarvester` processes the downloaded data. #### Interactions and Dependencies * `NihmsHarvester` is the central class, relying on configurations from `NihmsHarvesterConfig`, URL construction from `UrlBuilder` and `UrlType`, and data downloading from `NihmsHarvesterDownloader`. * `NihmsHarvesterCLI` and `NihmsHarvesterCLIRunner` are entry points for the application, primarily handling user interaction and delegation to the core `NihmsHarvester`. * `UrlBuilder` and `UrlType` ensure that the URLs used for data retrieval are correctly constructed and categorized. ### NIHMS Data Transform-Load Classes and Relationships #### Data Flow Overview * **Data Transformation and Loading**: * `NihmsTransformLoadCLIRunner` and `NihmsTransformLoadCLI` handle command-line interactions for data transformation and loading. * `NihmsTransformLoadService` transforms publication data and loads it using `SubmissionLoader`. * `PmidLookup` service to retrieve a PMID record from the [NBCI Entrez API](https://www.ncbi.nlm.nih.gov/books/NBK25497/). * **Data Transfer**: * `SubmissionDTO` encapsulates transformed submission data for transfer between components. * `NihmsPassClientService` service to provide interactions with the data via the PASS client. * `SubmissionLoader` loads the final submission data into PASS and accomplishes this by interfacing with the `NihmsPassClientService`. #### Interactions and Dependencies * `NihmsTransformLoadCLI` is the entry point and passes control to `NihmsTransformLoadCLIRunner`. * `NihmsTransformLoadCLIRunner` initializes the process using configurations from `NihmsTransformLoadConfig` and performs the transformation and loading by invoking `NihmsTransformLoadService`. * `NihmsTransformLoadConfig` provides necessary configuration settings to `NihmsTransformLoadService`. * `NihmsTransformLoadService` orchestrates the transformation of data using `NihmsPublicationToSubmission` and the loading of data using `SubmissionLoader`. * `NihmsPublicationToSubmission` is responsible for transforming PMC publication data and associated data to a `SubmissionDTO`, which is composed of `Grant`, `Publication`, `RepositoryCopy`, and `Submission` objects. * `SubmissionDTO` acts as a container for the transformed data. * `SubmissionLoader` is responsible for the final step of loading the data into PASS. ## Next Step / Institution Configuration Configuring the NIHMS loader to run at an institution requires several NIH/NLM accounts to be setup. The first step would be to familiarize yourself with the [PACM Guide](https://www.ncbi.nlm.nih.gov/pmc/utils/pacm/static/pacm-user-guide.pdf). Institutions that are universities typically have an Office of Research and that is a good starting point to finding out more information how PACR roles are assigned, typically a Signing Officer at the Office of Research can perform this function. Once access to PACM has been established, configuring the infrastructure where the Data Harvester and the Data Transform-Load applications is the next step. Since they are Java applications they can be scheduled as cron jobs on a server or run in the cloud. At Johns Hopkins University these applications are run by using [AWS Batch and ECS](/welcome-guide/deployment-architecture#pass-deployment-architecture). Optionally as a last step, setting up the [NIH Manuscript Submission](https://www.nihms.nih.gov) account would enable seeing which submissions have been made and how far along they are in the process. Once a submission makes its way through this process it appears in the PubMed Central data. This can be useful for troubleshooting records expected to be in the PubMed Central data. ### Manual NIHMS Token Generation Once a PACM account is established, generating the token can be done via the [PACM Website](http://www.pubmedcentral.nih.gov/utils/pacm/). Use the eRA Commons login and sign in with your username and password. Note that the login instructions may be different for every institution as the authentication mechanism may differ. After logging in, use the link at the top of the page that is labeled `API Token`. This page will generate a new token every time it is accessed and your new API token will appear here. A token is valid for three months, and currently there is no way of deleting a token, so each page access will create a new token. ### NIHMS Token Refresh The NIHMS harvester process requires an Authentication token. This token is available from the PACM Utils page and is valid for three months. There is currently no API available to refresh the token. In order to provide an automatic token refresh, there is a module within the NIHMS loader called `nihms-token-refresh` that performs an automatic refresh of the token and updates the AWS Parameter Store. More information about how this module works in a production environment can be found in the [Operation/Production Data Loaders section](/infrastructure-documenation/operations-production/ops-loaders#nihms-api-token-refresh-automation). ## Related Information The resources below will assist in setting up the accounts required to run the NIHMS Loader and understanding the submission process. * PubMed Central Resources * [PACM Guide](https://www.ncbi.nlm.nih.gov/pmc/utils/pacm/static/pacm-user-guide.pdf) * [PMC APIs](https://www.ncbi.nlm.nih.gov/pmc/tools/developers/#pmc-apis) * NIHMS Resources * [NIH Manuscript Submission](https://www.nihms.nih.gov) * [NIH Manuscript Submission Process](https://www.nihms.nih.gov/about/overview/) # Deposit Services Deposit Services (DS) facilitates the transfer of custodial content and metadata from end users to repositories,\ ensuring proper packaging, validation, and adherence to public specifications. The code for DS can be found in\ the [Pass Support](https://github.com/eclipse-pass/pass-support/tree/main/pass-deposit-services) repository. ## Summary Deposit Services are responsible for the transfer of custodial content and metadata from end users to repositories.\ End users transfer custody of their content to PASS by performing a submission through the HTML user interface, and\ Deposit Services subsequently transfers the custody of content to downstream repositories. Custody transfer goes beyond the successful transfer of bytes. The package content may be validated (accepted or rejected)\ by the downstream repository, and this validation may occur asynchronously with respect to the transfer of bytes.\ Material consisting of one or more files from the end user submission is sent to the downstream repository in a single\ file known as the "package". A package can be thought of as a zip file containing user submitted files plus metadata generated by DS. Public\ specifications govern the structure of the zip file and the kind of metadata provided: * BagIT * NIHMS bulk submission spec In the context of documenting inputs and outputs, DS uses the public PASS model as input and output, and produces a\ package which adheres to a public specification and downstream repository requirements. The internal Deposit Service\ Model described below is never persisted, serialized, or provided to a caller. It is entirely internal to the Deposit\ Service. DS is a backend service component in PASS written in the Java language and using the[Spring Boot](https://spring.io/projects/spring-boot) framework. It has no user-facing elements. DS reacts asynchronously to `SubmissionMessage` and`DepositMessages` messages emitted by the Pass-Core component. ## Submission/Deposit Logical Flow
Submission & Deposit Logical Flow

Submission & Deposit Logical Flow

1. Listener waits for "Submission Complete" message. Invokes the Builder. 2. Builder retrieves Submission and metadata from the Archive, instantiates internal Deposit Services model. 3. Listener invokes the Assembler. The Assembler: * Determines archive format (tar.gz, zip) * Builds content manifest for the archive * Uses a package provider to create the content stream 4. Package Provider adds package-specific metadata according to downstream repository requirements. 5. Transport is selected, establishes a session with remote archive, and streams the package. 6. Later, a Deposit Status Processor (not shown) will update the Deposit status, confirming custody transfer. The Deposit\ Status Processor is optional. In this guide we step through various topics on Deposit Services: * [Knowledge Needed / Skills Inventory](/developer-documentation/deposit-service/ds-know-need) * [Technologies Utilized](/developer-documentation/deposit-service/ds-tech-util) * Technical Deep Dive * [Model](/developer-documentation/deposit-service/ds-model) * [Statuses](/developer-documentation/deposit-service/ds-status) * [Business Logic](/developer-documentation/deposit-service/ds-business) * [Assemblers](/developer-documentation/deposit-service/ds-assemblers) * [Configuration](/developer-documentation/deposit-service/ds-configuration) * [Next Steps / Institution Configuration](/developer-documentation/deposit-service/ds-new-institution) # Knowledge Needed / Skills Inventory The knowledge needed to comprehend the DS documentation includes a general understanding of the PASS submission workflow and processing of deposits into remote repositories. Understanding the major components of PASS and the communication channels between the components is also valuable for this document. From a technical perspective, the following are recommended: * [Java 17+](https://www.oracle.com/java/technologies/downloads/) * [Spring Boot](https://spring.io/projects/spring-boot) * Messaging * IO Programming * A general understanding of service components from a Systems Design perspective * An understanding of system integration is important because of the nature of remote repository deposits. # Technologies Utilized The code for DS can be found in the [pass-support repository](https://github.com/eclipse-pass/pass-support/tree/main/pass-deposit-services). DS is a backend service that is written in [Java](https://www.java.com/en/) and [Spring Boot](https://spring.io/projects/spring-boot). The DS project is a [Maven](https://maven.apache.org/) project, which is used to execute the standard lifecycle tasks for software development (i.e. build/test/package/release) of the DS service. The Maven POM is a child of the [`eclipse-pass/pass-support` POM](https://github.com/eclipse-pass/pass-support) which is a child of the [`eclipse-pass/main` POM](https://github.com/eclipse-pass/main). Here are the most significant technologies used in DS: * [Spring Boot](https://spring.io/projects/spring-boot) is used for functionality such as listening for JMS messages, configuration, “wiring-up” of DS components via dependency injection, and tests. * There are unit and integration tests in DS. Tests are executed using [JUnit](https://junit.org/junit5/) and Spring Boot Test. [TestContainers](https://testcontainers.com/) are used for integration tests with pass-core. * [Docker](https://www.docker.com/) is used for building a DS docker image that is used for deployment. # Model The Deposit Services data model describes the interaction between Deposit Services and the PASS data model, detailing\ how various resources, such as `Submission`, `Repository`, `Deposit`, and `RepositoryCopy` are managed. Additionally, it\ outlines the internal data model, configuration, packaging, and transport mechanisms used by Deposit Services to\ facilitate the transfer and validation of submissions to downstream repositories. ## PASS Model Deposit Services uses objects in the PASS data model, which are distinct from the Deposit Services' internal model.\ Objects in the PASS data model are persisted in the PASS core service. Thus, any interaction\ with PASS resources will require CRUD operations (using the PassClient) by Deposit\ Services. PASS objects used by Deposit Services are: * `Submission`: Read by Deposit Services, updates `Submission.AggregatedDepositStatus`. * `Repository`: Only ever read by Deposit Services, never modified. * `Deposit`: Created and modified by Deposit Services. * `RepositoryCopy`: Created and modified by Deposit Services. Note that the NIHMS loader also creates`RepositoryCopy` resources.
Deposit Services PASS Model

Deposit Services PASS Model

Each `Submission` resource links to one or more `Repository` resources. Deposit Services will create a `Deposit`\ and `RepositoryCopy` resource for each `Repository` linked to the `Submission`. Deposit Services will attempt to deposit\ to the downstream system represented by the `Repository`. The status of a deposit to a downstream repository will be\ recorded on the `Deposit` resource. That is to say, the `Deposit` records the transaction and its success or failure\ with a `Repository`, and the `RepositoryCopy` records where the Repository stored the content of the `Deposit`. ## Deposit Services Internal Model Deposit Services has an internal object model that is distinct from the PASS data model. Instances of the internal\ object model are not persisted in the PASS repository, or anywhere else. Upon receipt of a `Submission` (i.e. the\ external PASS model), Deposit Services immediately converts it to an instance of the internal model using a Model\ Builder.
Deposit Services Internal Model

Deposit Services Internal Model

### Deposit Model * `DepositSubmission`: internal representation of a PASS Submission. * `DepositMetadata`: metadata describing the submission, parsed from the "metadata blob" (**`Submission.metadata`**)\ and other `Submission` properties. * `DepositSubmission` is the central entity in the internal DS model. It brings entities and properties of the public PASS\ model into a model specific to producing a package. Many of the fields or classes are bibliographic in nature, with the`DepositFile` linking to the binary content of the submission (the files uploaded by the end user in the submission\ process). A secondary purpose of the internal DS model is to surface bibliographic metadata explicitly, since a\ primary responsibility of DS is to map bibliographic metadata to package metadata. Hard-coding bibliographic metadata\ in the internal model could be considered an anti-pattern. ### Configuration Model * `Packager`: encapsulates configuration of the `Assembler` and `Transport` for every\ downstream repository in `repositories.json`. Each repository configured in `repositories.json` should to reference\ a `Repository` resource in the PASS repository. * `RepositoryConfig`: Java representation of a single repository configuration in `repositories.json`. The\ configuration for a repository includes directives for the transport protocol used for deposit (including\ authentication credentials), packaging specification used for deposit, and packaging options.
Deposit Service Configuration Model

Deposit Service Configuration Model

* Each downstream repository is represented in the public PASS model as a `Repository`; each `Repository` carries a unique\ key, which is a short, human-readable string (e.g. `jscholarship`, `dash`, `pmc`). Each `Repository.key` is represented in\ the DS configuration model as `RepositoryConfig.repositoryKey`. When a Submission is processed, the configuration for the\ Repository is resolved by its key. ### Packaging Model * `Assembler`: responsible for creating and streaming the content (i.e. the files uploaded by the end-user and any\ metadata required by the packaging specification) of a Submission according to a packaging specification. * `PackageStream`: the content of a `Submission` to be deposited to a downstream repository as a stream, as opposed\ to bytes held in a buffer or stored on a file system. * `Transport`: an abstraction representing the physical protocol used to transfer the package stream from the PASS\ repository to the downstream repository. ### Messaging Model * `CriticalRepositoryInteraction`: CRI for short. Performs an optimistic locking (`If-Match` using an Etag) "critical"\ modification on a PASS resource, with a built-in retry mechanism when a modification fails. Each CRI has a\ pre-condition, critical section, and post-condition. The pre-condition must be met before the critical section is\ executed. The post-condition determines whether the application of the critical section was successful. The built-in\ retry mechanism re-uses the pre/post/critical functions in the case of a conflict. ## Model Builder Upon receipt of a Submission, the `DepositSubmissionModelBuilder` is invoked to produce an instance\ of `DepositSubmission`. The `DepositSubmissionModelBuilder` accepts a `Submission` ID for conversion to`DepositSubmission`. ## Assembler Responsible for assembling the content of a submission into a streamable package (i.e. the `Assembler` returns\ a `PackageStream` instance). This includes: * Resolving the custodial content being deposited from the PASS repository. * Generating any metadata required by the packaging specification. * Generating any metadata required by the downstream repository. * Encapsulating all of the above into a stream of bytes that meets a packaging specification. Implementing the Configurable Metadata Framework focuses on the support of pluggable Assemblers within Deposit\ Services; different `Assembler` implementations can include metadata required for their repository. ### Custodial and supplemental resources The term *custodial resource* is used throughout: a custodial resource is content that was uploaded by the end user for\ deposit to a downstream repository, such as their data sets, manuscripts, etc. Non-custodial resources (i.e. *supplemental*\ *resources*) include metadata describing the content, for example, BagIt tag files or DSpace METS XML files. #### Abstract Assembler and Archiving Package Stream There are two abstract classes to help developers create `Assembler` implementations. The `AbstractAssembler`\ and `ArchivingPackageStream`. There is also a concrete class named `SimplePackageStream` that can be used for\ non-archive integrations. The `AbstractAssembler` contains shared logic for building a list of custodial resources to be deposited. Concrete\ implementations accept the list of custodial resources (among other parameters, including the packaging specification)\ and produce the `PackageStream`. The `ArchivingPackageStream` contains shared logic for assembling multiple files into a single zip, tar, or tar.gz file. The `SimplePackageStream` contains the associated `DepositSubmission`, List of `DepositFileResources`, and returns\ metadata to be sent to repository. #### MetadataBuilder and PackageStream.Metadata `PackageStream.Metadata` is an interface that provides package-level metadata. The `MetadataBuilder` is a fluent API for\ creating physical package-level metadata, such as: * Packaging specification - a URI identifying the package specification used. * Package size (bytes) and its checksum. * The package name. * Mime type, compression used, and archive format. The `PackageStream.metadata()` method returns the `PackageStream.Metadata` for a `PackageStream` instance. Because some\ metadata is unknown prior to streaming (e.g. the package size), the metadata returned by this method may be incomplete\ until after the stream has been read. #### ResourceBuilder and PackageStream.Resource `PackageStream.Resource` is an interface that provides metadata describing a resource within the\ package. `ResourceBuilder` is a fluent API for creating physical metadata describing each resource (i.e. file) within\ the package: * File size (bytes) * Filename, including its path relative to the package root * Checksum and mime type The `PackageStream.resources()` method answers an `Iterator` over each `Resource` in the `PackageStream`. #### Package Provider `PackageProvider` is an interface that is invoked by Deposit Services when a `PackageStream` is streamed to a repository\ via a `Transport`. `PackageProvider` represents a streaming lifecycle interface that has three methods: `start(...)`, `packagePath(...)`,\ and `finish(...)`. The `start(...)` method is invoked after the custodial resources have been assembled, but before\ streaming has started. The `packagePath(...)` method is invoked prior to streaming each custodial resource.\ The `finish(...)` method is invoked after all the custodial resources have been streamed, and provides an opportunity\ for the `PackageProvider` to add supplemental resources to the package being streamed. For example, a BagIt `PackageProvider` would ensure that each custodial resource is pathed under `/data`\ when implementing `packagePath(...)`. After the custodial resources are streamed, the BagIt `PackageProvider` would\ assemble and stream all the BagIt metadata: bagit.txt and any other tag files. ## Transport Responsible for transferring the bytes of a package (i.e., a `PackageStream`) to an endpoint. The Transport API is\ designed to support any transport protocol. Each downstream repository in `repositories.json` must be configured with\ a `Transport` implementation. The `Assembler` and the `PackageProvider` create the package, and the `Transport` is the "how" of how a package is\ transferred to a downstream repository. Choosing the `Transport` to be used depends on the support of the downstream\ repository for things like SFTP, DSpace API (HTTP), InvenioRDM (HTTP) or other protocols. For example, a BagIt `Assembler` and `PackageProvider` would produce BagIt packages. Those packages may be transported\ to downstream repositories using SFTP or a custom Transport implementation. The `Transport` to be used is a\ matter of configuration in `repositories.json`. ### SFTP Supports the transport of the package stream using SFTP. #### DSPACE API Supports the transport of the package stream that will be sent to a DSpace repository. This implementation uses\ HTTP to call the DSpace REST API. #### InvenioRDM Supports the transport of the package stream that will be sent to a InvenioRDM repository. This implementation uses\ HTTP to call the InvenioRDM REST API. # Statuses Deposit Services primarily acts on three types of resources: `Submission`, `Deposit`, and `RepositoryCopy`. Each of\ these resources carries a status. Managing and reacting to the values of resource statuses is a major function of\ Deposit Services. Abstractly, Deposit Services categorizes the value of any status as either *intermediate* or *terminal*. > It isn't clear, yet, whether this abstract notion of *intermediate* and *terminal* need to be shared amongst\ > components of PASS. If so, then certain classes and interfaces in the Deposit Services code base should be extracted\ > into a shared component. A *terminal* status means the resource has completed its workflow and reached the end. No additional state changes are\ expected, and the resource is considered final and read-only. An *intermediate* status means the resource is still in progress within its workflow and has not yet reached completion.\ The resource is expected to be modified until it achieves a *terminal* status. A general pattern within Deposit Services is that resources with *terminal* status are explicitly accounted for (this is\ largely enforced by *policies* which are documented elsewhere), and are considered "read-only". ## Submission Status Submission status is enumerated in the `AggregatedDepositStatus` class. Deposit services considers the following values: * `NOT_STARTED` (*intermediate*): Incoming Submissions from the UI must have this status value. * `IN_PROGRESS` (*intermediate*): Deposit services places the Submission in an `IN_PROGRESS` state right away. When a\ thread observes a `Submission` in this state, it assumes that *another* thread is processing this resource. * `FAILED` (*intermediate*): Occurs when a non-recoverable error happens while processing the `Submission`. * `ACCEPTED` (*terminal*): Deposit services places the Submission into this state when all of its `Deposit`s have\ been `ACCEPTED`. * `REJECTED` (*terminal*): Deposit services places the Submission into this state when all of its `Deposit`s have\ been `REJECTED`. ## Deposit Status Deposit status is enumerated in the `DepositStatus` class. Deposit services considers the following values: * `SUBMITTED` (*intermediate*): the custodial content of the `Submission` has been successfully transferred to\ the `Deposit`s `Repository`. * `ACCEPTED` (*terminal*): the custodial content of the `Submission` has been accessioned by the `Deposit`'s `Repository`\ , i.e. custody of the `Submission` has successfully been transferred to the downstream `Repository`. * `REJECTED` (*terminal*): the custodial content of the `Submission` has been rejected by the `Deposit`'s `Repository`,\ i.e. the downstream `Repository` has refused to accept custody of the `Submission` content. * `FAILED` (*terminal*): the transfer of custodial content to the `Repository` failed, or there was some other error\ updating the status of the `Deposit`. * `RETRY` (*intermediate*): the downstream repository was not reachable. The Deposit will be retried at a later time. ## RepositoryCopy Status RepositoryCopy status is enumerated in the `CopyStatus` class. Deposit services considers the following values: * `COMPLETE` (*terminal*): a copy of the custodial content is available in the `Repository` at this location. * `IN_PROGRESS` (*intermediate*): a copy of the custodial content is *expected to be* available in the `Repository` at\ this location. The custodial content should not be expected to exist until the `Deposit` status is `ACCEPTED`. * `REJECTED` (*terminal*): the copy should be considered to be invalid. Even if the custodial content is made available\ at the location indicated by the `RepositoryCopy`, it should not be mistaken for a successful transfer of custody. RepositoryCopy status is dependent on the `Deposit` status. They will always be consistent. For example,\ a `RepositoryCopy` cannot be `COMPLETE` if the Deposit is `REJECTED`. If a `Deposit` is `REJECTED`, then the`RepositoryCopy` must also be `REJECTED`. ## Common Permutations There are some common permutations of these statuses that will be observed: * `ACCEPTED` `Submission`s will only have `Deposit`s that are `ACCEPTED`. Each `Deposit` will have\ a `COMPLETE` `RepositoryCopy`. * `REJECTED` `Submission`s will only have `Deposit`s that are `REJECTED`. `REJECTED` `Deposit`s will not have\ any `RepositoryCopy` at all. * `IN_PROGRESS` `Submission`s may have zero or more `Deposit`s in any state. * `FAILED` `Submission`s should have zero `Deposit`s. * `ACCEPTED` `Deposit`s should have a `COMPLETE` `RepositoryCopy`. * `REJECTED` `Deposit`s will have a `REJECTED` `RepositoryCopy`. * `SUBMITTED` `Deposit`s will have an `IN_PROGRESS` `RepositoryCopy`. * `FAILED` `Deposit`s will have no `RepositoryCopy`. # Business Logic Deposit Services system processes messages concurrently from both deposit and submission queues, utilizing dedicated\ listeners for each queue. This section outlines the workflow, detailing how messages are handled, processed, and how the\ corresponding resources are managed and updated throughout the deposit lifecycle. ## Message flow and Business Services Each message listener, one each for the `deposit` and `submission` queues, can process messages concurrently. The `submission` queue is processed by the `SubmissionListener`, which resolves the `Submission` resource represented\ in the message, and hands off processing to the `SubmissionProcessor`. The `SubmissionProcessor` builds\ a `DepositSubmission`, which is the Deposit Services' analog of a `Submission` containing all the metadata and\ custodial content associated with a `Submission`. After building the `DepositSubmission`, the processor calls the`DepositTaskHelper` to perform the actual deposit. The `DepositTaskHelper` delegates the deposit steps to an instance\ of a `DepositTask`. Importantly, the `SubmissionProcessor` updates the `Submission` resource in the repository as\ being *in progress*. The `DepositTask` class contains the primary logic for packaging, streaming, and verifying the transfer of content from\ the PASS repository to downstream repositories. The `DepositTask` will determine if the transfer of custodial content\ has succeeded, failed, or is indeterminable (i.e. an asynchronous deposit process that has not yet concluded). The\ status of the `Deposit` resource associated with the `Submission` will be updated accordingly. ## Failure Handling A *failed* `Deposit` or `Submission` is marked with `Deposit.DepositStatus = FAILED` or `Submission.AggregateDepositStatus = FAILED`.\ When a resource has been marked `FAILED`, Deposit Services will ignore any messages relating to the resource. A resource will be considered as failed when errors occur during the processing of `Submission` and `Deposit` resources.\ Some errors may be caused by transient network issues, or a server being rebooted. In the case of such failures,\ The `Deposit.depositStatus` will be set to `RETRY`, and Deposit Services will retry for n number of days after the`Submission` is created. The number of days is set in an application property named `pass.status.update.window.days`. `Submission` resources are failed when: 1. Failure to build the Deposit Services model for a Submission. 2. There are no files attached to the Submission. 3. Any file attached to the Submission is missing a location URI (the URI used to retrieve the bytes of the file). 4. An error occurs saving the state of the `Submission` in the repository (arguably a transient error). For more details, refer to the `SubmissionProcessor`. Right now, when a `Submission` is failed, manual intervention may be required.\ Deposit Services does retry the failed `Deposit` resources of the `Submission`. However, some of the failure scenarios\ above must be resolved by the user. It is possible the end-user will need to re-create the submission in the user\ interface, and resubmit it. `Deposit` resources are failed when: 1. An error occurs building a package. 2. An error occurs streaming a package to a `Repository` (potentially transient). 3. An error occurs polling (potentially transient) or parsing the status of a `Deposit`. 4. An error occurs saving the state of a `Deposit` in the repository (again, potentially transient). See `DepositTask` for details. Deposits fail for transient reasons; a server being down, an interruption in network\ communication, or invalid credentials for the downstream repository are just a few examples. As stated, DS will retry\ failed `Deposit` resources for n number of days after the creation of the associated `Submission`. The number of days\ is set in an application property named `pass.status.update.window.days`. ## Spring Error Handler Certain Spring sub-systems like Spring MVC, or Spring Messaging, support the notion of a "global" `ErrorHandler`.\ Deposit Services provides an implementation **`DepositServicesErrorHandler`**, and it is used to catch exceptions thrown\ by the `DepositListener`, `SubmissionListener`, and is adapted as a `Thread.UncaughtExceptionHandler` and\ as a `RejectedExecutionHandler`. Deposit Services provides a `DepositServicesRuntimeException` (`DSRE`), which has a field `PassEntity resource`.\ If the `DepositServicesErrorHandler` catches a `DSRE` with a non-`null` resource, the error handler will test the type\ of the resource, mark it as failed, and save it in the repository. In essence: `Deposit` and `Submission` resources will be marked as failed if a `DepositServicesRuntimeException` is\ thrown from one of the JMS processors, or from the `DepositTask`. As a developer, if an exceptional condition does**not** warrant a failure, then do not throw `DepositServicesRuntimeException`. Instead, consider logging a warning or\ throwing a `DSRE` with a `null` resource. Likewise, to fail a resource, all you need to do is throw a `DSRE` with a\ non-`null` resource. The `DepositServicesErrorHandler` will do the rest. Since the state of a resource can be modified at any time by any actor in the PASS infrastructure, the`DepositServicesErrorHandler` encapsulates the act of saving the failed state of a resource within a `CRI`. The*pre-condition* for updating the resource is that it must *not* be in a *terminal* state. For example, if the error\ handler is updating the state from `SUBMITTED` to `FAILED`, but another actor has modified the state of the resource to`REJECTED` in the interim, the *pre-condition* will fail. Modifying the state of a resource after it reaches its*terminal* state is not logical. In conclusion, the `DepositServicesErrorHandler` will not mark a resource as failed\ if it is in a *terminal* state. ## Spring Boot Context Deposit Services is implemented using Spring Boot, which heavily relies on Spring-based annotations and conventions to\ create and populate a Spring `ApplicationContext`, arguably the most important object managed by the Spring runtime.\ If you are unfamiliar with said annotations and conventions, the following resources would be beneficial to learning more\ about them: * [Baeldung](https://www.baeldung.com/spring-application-context) * [Spring Boot Docs](https://docs.spring.io/spring-boot/documentation.html) The entrypoint into the Deposit Services is the `DepositApp` class. Spring beans are created entirely in Java code by the`DepositConfig` and `JmsConfig` classes. ## Build and Deployment Deposit Services' primary artifact is a single self-executing jar. In the PASS infrastructure, the Deposit Services\ self-executing jar is deployed inside a simple Docker container. Deposit Services can be built by running the following command: ```shell mvn clean install ``` The main Deposit Services deployment artifact is located in `deposit-core/target/pass-deposit-service-exec.jar`. It is\ this jar file that is included in the [Docker image for Deposit Services](https://github.com/eclipse-pass/pass-support/pkgs/container/deposit-services-core),\ and posted on the [GitHub Release page](https://github.com/eclipse-pass/pass-support/releases). # Assemblers Developing a package provider primarily deals with extending or re-using `Assembler`-related abstract classes and\ implementations, but it is helpful to understand the context in which `Assembler`s operate. Assemblers are responsible for\ gathering custodial and supplemental resources associated with a submission and returning a `PackageStream` of those\ resources according to a packaging specification. Deposit Services will then stream the package to a downstream\ repository via a `Transport`. ## Use Case An `Assembler` implementation is required for every packaging specification you wish to support. For example, if you\ want to produce [BagIt packages](https://tools.ietf.org/html/rfc8493) and [DSpace METS packages](https://wiki.duraspace.org/display/DSPACE/DSpaceMETSSIPProfile),\ you would need two `Assembler` implementations, each responsible for producing packages that align with their respective\ specifications. Another reason to develop an `Assembler` is to control how the metadata of a submission is mapped into your package. For\ example, if your DSpace installation requires custom metadata elements, you would need to develop or extend an\ existing `Assembler` to include the custom metadata as appropriate to your environment, by way of implementing a custom\ Package Provider. One easy approach would be by extending a base `Assembler` class without having to write something entirely new. ## Quick Start 1. Create your Assembler class that extends `org.eclipse.pass.deposit.assembler.AbstractAssembler` 2. Create your Package Provider class that\ implements `org.eclipse.pass.deposit.assembler.PackageProvider` To get started with testing:\ Create your package verifier that implements `org.eclipse.pass.deposit.assembler.PackageVerifier`\ Extend and implement `org.eclipse.pass.deposit.assembler.AbstractThreadedAssemblyIT` ## API Overview ### Assembler API The main entrypoint into the Assembler API is on the `Assembler` interface:`PackageStream assemble(DepositSubmission, Map)`\ where the `DepositSubmission` is the internal representation of a `Submission`, and the `Map` is a set of package\ options read from `repositories.json`. The `AbstractAssembler` provides an implementation of `assemble(DepositSubmission, Map)`, and requires its subclasses to\ implement: `PackageStream createPackageStream(DepositSubmission, List, MetadataBuilder, ResourceBuilderFactory, Map)` Where the `List` is the custodial content of the submission, the `MetadataBuilder` allowing\ modification of the package-level metadata, and the `ResourceBuilderFactory` used to generate an instance\ of `ResourceBuilder` for each `DepositFileResource`. The primary benefit of extending `AbstractAssembler` is that the logic for identifying the custodial resources in the\ submission and creating their representation as `List` is shared. Subclasses of `AbstractAssembler`\ must instantiate and return a `PackageStream`. Examples can be found at: `DspaceAssembler`, `NihmsAssembler`, `InvenioRdmAssembler`, and `BagItAssembler` ### PackageStream API Assemblers are invoked by Deposit Services and return\ a `PackageStream`. The `PackageStream` represents the content to be sent to a downstream repository. Conceptually,\ the `PackageStream` behaves like a Java `InputStream`: the bytes for the stream can come from anywhere (memory, a\ file on disk, or retrieved from another network resource), and can generally only be read once. Practically, the `PackageStream` represents an archive file: either a ZIP, TAR, or some variant like TAR.GZ. This is\ encapsulated by the `ArchivingPackageStream` class. Re-using the `ArchivingPackageStream` class has the advantage\ that your package resources will be bundled up in a single archive file according to the options supplied to\ the `Assembler` (e.g. compression and archive type to use). To instantiate an `ArchivingPackageStream` class requires an instance of `PackageProvider`. There is also a `SimplePackageStream` class that contains the associated `DepositSubmission`, List of`DepositFileResources`, and metadata to be sent to repository. This class may be used in integrations where the\ individual file resources are needed for processing the repository deposit. For example, the InvenioRDM integration is\ one such repository. ### PackageProvider API The `PackageProvider` interface was developed as an ad hoc lifecycle for streaming a package: there's a `start(...)`\ and `finish(...)` method, along with a `packagePath(...)` method. `PackageProvider` also defines a new interface:`SupplementalResource`. This interface is returned by the `finish(...)` method, allowing the `PackageProvider`\ implementation to generate supplemental (i.e. BagIt tag files or METS.xml files) content after the rest of the package\ has been streamed. Implementing this interface therefore allows for customizing where resources will appear in the package, and to\ customize the metadata that appears in the package. Because packaging specifications generally have something to say about what resources are included where in the package,\ a Package Provider is loosely coupled to a package specification. For example, a Package Provider that placed custodial\ resources in the `/foo` directory would be incompatible with a BagIt packaging specification, which\ requires custodial resources to appear under `/data`. Similarly, if your Package Provider is to align\ with a DSpace METS packaging scheme, it will need to produce a `/METS.xml` file with the required content.\ Therefore, any `PackageProvider` implementation can be used with any `Assembler` implementation as long as the package\ specification shared between the two is not violated. ## Assembler Development Recap Implementations of `AbstractAssembler` that return a single archive for deposit will return an `ArchivingPackageStream`\ which uses a `PackageProvider` to path resources and generate supplemental metadata contained in the package. For\ integrations that process each file resource, the `AbstractAssembler` implementation should return `SimplePackageStream`\ that can be used later by the `Transport` for processing. *When developing your own `Assembler` that returns a `ArchivingPackageStream`, you will need to:* * Extend `AbstractAssembler` * Implement `PackageProvider`, including the logic to produce supplemental package content like BagIt tag files or\ DSpace METS.xml files * Construct `ArchivingPackageStream` with your `PackageProvider` and return that from your `AbstractAssembler`\ implementation Examples of implemented package providers: * `NihmsPackageProvider` * `BagItPackageProvider` *When developing your own `Assembler` that returns a `SimplePackageStream`, you will need to:* * Extend `AbstractAssembler` * Construct `SimplePackageStream` return that from your `AbstractAssembler` implementation Examples of implemented such assemblers: * `InvenioRdmAssembler` ## Concurrency Assemblers exist in the Deposit Services runtime as singletons. A single `Assembler` instance may be invoked from\ multiple threads, therefore all the code paths executed by an `Assembler` must be thread-safe. `AbstractAssembler` and `ArchivingPackageStream` are already thread-safe; your concrete implementation\ of `AbstractAssembler` and `PackageProvider` will need to maintain that thread safety. Streaming a package inherently\ involves maintaining state, including the updating of metadata for resources as they are streamed. One strategy for maintaining thread safety is to scope any state maintained over the course of streaming a package to\ the executing thread. `Assembler` implementations are free to use whatever mechanisms they wish to ensure thread\ safety, but Deposit Services accomplishes this in its codebase by simply instantiating a new instance of\ state-maintaining classes each time the `Assembler.assemble(...)` is invoked, and ensures that state is not shared (i.e.\ kept on the Thread stack and not in the JVM heap). For example: * `AbstractAssembler` instantiates a new `MetadataBuilder` each time using a factory pattern. * `AbstractAssembler` implementations instantiate a new `ArchivingPackageStream` each time. * `DefaultStreamWriterImpl` instantiates a new `ResourceBuilder` for each resource being streamed using a factory\ pattern. The factory objects may be kept in shared memory (i.e. as instance member variables), but the objects produced by the\ factories are maintained in the Thread stack (as method variables). After a `PackageStream` has been opened and\ subsequently closed, these objects will be released and garbage collected by the JVM. To help ensure thread safety,\ there is an integration test fixture, `ThreadedAssemblyIT`, which can be subclassed and used by `Assembler` integration\ tests to verify thread safety. ## Testing Adequate test coverage of `Assemblers` includes proper unit testing. This document presumes that you've adequately unit\ tested your implementation, and instead focuses on integration testing. Integration testing of `Assemblers` is supported by some shared test fixtures in the core Deposit Services codebase. ### ThreadedAssemblyIT The approach taken by the shared `ThreadedAssemblyIT` is to invoke the `Assembler` under test directly using\ random `DepositSubmission`s. A singleton `Assembler` implementation under test is retrieved from the IT subclass that\ you provide a number of different `DepositSubmission`s are used to concurrently invoke `Assembler.assemble(...)` on the\ singleton instance under test the `PackageStreams` returned by the `Assembler` under test are streamed to and stored on\ the filesystem. A package verifier supplied by the IT subclass verifies the content of the packages. The advantage of extending `ThreadedAssemblyIT` is that it ensures that your `Assembler` can be invoked concurrently by\ multiple threads while avoiding the complexity of setting up and configuring the Deposit Services runtime. The Spring\ Framework is not used, the Deposit Services runtime is not required, and no Docker containers are needed: the IT is\ simple Java and JUnit. The downside is that your full runtime is not being integration tested, only your `Assembler`. To use `ThreadedAssemblyIT`, extend it, and implement the required methods: * `assemblerUnderTest()`: provide an AbstractAssembler instance, fully initialized and ready to be invoked * `packageOptions()`: provides a set of package options, used when creating the PackageStream and storing it on disk.\ The package options include: * The package specification to be used * The compression algorithm used when creating the package * The checksumming algorithm to be used when calculating package and package resource checksums * `packageVerifier()`: answers a `PackageVerifier` which inspects a package stored on the filesystem and verifies its\ content. You must implement a `PackageVerifier` for each `Assembler` being tested. The test logic automatically executes in `ThreadedAssemblyIT.testMultiplePackageStreams()`. The `PackageVerifier` is\ very important: it does most of the heavy lifting with respect to passing or failing the integration test, so it must be\ well written and test all aspects of a generated package. Examples of these ITs: * `BagItThreadedAssemblyIT` * `NihmsThreadedAssemblyIT` ### PackageVerifier Each `Assembler` that is developed should have a corresponding `PackageVerifier`. The `PackageVerifier` is the primary\ interface for verifying that a package written to disk contains the expected content. The primary method to implement\ is:`void verify(DepositSubmission, ExplodedPackage, Map)` where the `DepositSubmission` is the original\ submission, `ExplodedPackage` is the generated package on disk, and the `Map` includes the options supplied to\ the `Assembler` that created the package. The verifier is responsible for: * Ensuring that every custodial file from the submission is present and accounted for in the package. * Ensuring there are no extraneous custodial files in the package that are not in the submission. * Ensuring that the custodial files checksums are correct. * Ensuring that the proper supplemental files are present in the package and have the correct content. Essentially all aspects of a generated package must be verified through a `PackageVerifier`. The `PackageVerifier` interface includes a helper method `verifyCustodialFiles` for ensuring that there is a\ custodial file in the package for each submitted file, and that there are no unexplained custodial files present in the\ package.`void verifyCustodialFiles(DepositSubmission, File, FileFilter, BiFunction)`\ where `DepositSubmission` is the original submission, the `File` is the directory on the filesystem that contains the\ exploded package, the `FileFilter` selects custodial files from the package directory, and the `BiFunction` accepts\ a `DepositFile` from the submission and maps it to its expected location in the package directory. Examples: * `NihmsPackageVerifier` ## Runtime Deposit Services is a Spring Boot application, and `Assembler`s are simply a component executed within the application.\ If you are familiar with Spring and/or Spring Boot, you are welcome to leverage its features as you wish. Regardless of\ your views of Spring, you need to be aware of Spring in these cases: * When extending `SubmitAndValidatePackagesIT` your IT will need to use the `SpringRunner` * Your `Assembler` implementation must be annotated with `@Component` ### Wiring So, how is your `Assembler`, `PackageStream`, and `PackageProvider` wired together? As outlined above, the wiring of\ these components is straightforward. You can either "hardwire" your implementations at compile-time, or you can leverage\ Spring dependency injection. Deposit Services uses Spring Auto Configuration to discover your `Assembler` on the classpath on boot. Supporting Spring\ Auto Configuration is very simple, by ensuring that your `Assembler` implementation is annotated with `@Component`. # Configuration The primary mechanism for configuring Deposit Services is through environment variables. This aligns with the patterns used in development and production infrastructure which rely on Docker and its approach to runtime configuration. Secondary configuration is provided by Spring Boot `application.properties`. This configuration includes lower-level parameters such as message queues, the base URL of Pass Core, etc. ## NIHMS Credentials Configuration In order for PASS to be able to make deposits into NIHMS and monitor for status updates, there are credentials that need to be created in NIHMS. Please review the [NIHMS Credentials Configuration](https://github.com/eclipse-pass/pass-documentation/blob/main/infrastructure-documenation/operations-production/ops-nihms.md) for the details. ## Production Configuration Variables | Environment Variable | Default Value | Description | | --------------------------------------- | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DSPACE_API_URL` | | Base URL for DSpace API | | `DSPACE_WEBSITE_URL` | | Base URL for DSpace website | | `DSPACE_SERVER` | | DSpace server hostname:port | | `DSPACE_COLLECTION_HANDLE` | | Handle for collection where deposits will be put | | `DSPACE_USER` | | DSpace user | | `DSPACE_PASSWORD` | | DSpace password | | `INVENIORDM_API_BASE_URL` | | Base URL for InvenioRDM API | | `INVENIORDM_VERIFY_SSL_CERT` | true | Required since the localhost InvenioRDM runs with a self-signed certificate | | `INVENIORDM_API_TOKEN` | | InvenioRDM API token | | `PMC_FTP_HOST` | | IP address or host name of the NIH SFTP server | | `PMC_FTP_PORT` | | TCP control port of the NIH SFTP server | | `PMC_FTP_USER` | | PMC SFTP user | | `PMC_FTP_PASSWORD` | | PMC SFTP password | | `PASS_DEPOSIT_QUEUE_SUBMISSION_NAME` | submission | Name of the JMS queue that has messages pertaining to `Submission` resources | | `PASS_DEPOSIT_QUEUE_DEPOSIT_NAME` | deposit | Name of the JMS queue that has messages pertaining to `Deposit` resources | | `PASS_DEPOSIT_REPOSITORY_CONFIGURATION` | classpath:/repositories.json | Points to a json file containing the configuration for the transport of custodial content to remote repositories. Values must be [Spring Resource URIs](https://docs.spring.io/spring-framework/reference/core/resources.html#resources-implementations). | | `PASS_CORE_URL` | | URL used to communicate with the PASS Core API. Normally this variable does not need to be changed. | | `PASS_CORE_PASSWORD` | | Password used for `Basic` HTTP authentication to the PASS Core API | | `PASS_CORE_USER` | | Username used for `Basic` HTTP authentication to the PASS Core API | | `NIHMS_MAIL_HOST` | | Host URL of the email service that will receive NIHMS emails regarding deposit statuses. | | `NIHMS_MAIL_PORT` | | Port of the email service that that will receive NIHMS emails regarding deposit statuses. | | `NIHMS_MAIL_USERNAME` | | Email address that will receive the NIHMS emails regarding deposit statuses. | | `NIHMS_MAIL_PASSWORD` | | Password of the email address that will receive the NIHMS emails regarding deposit statuses. | | `NIHMS_MAIL_TENANT_ID` | | The tenant ID if the `NIHMS_MAIL_HOST` is a cloud provided email services (e.g. Office 365). | | `NIHMS_MAIL_CLIENT_ID` | | The client ID if the `NIHMS_MAIL_HOST` is a cloud provided email services (e.g. Office 365). | | `NIHMS_MAIL_CLIENT_SECRET` | | The client secret if the `NIHMS_MAIL_HOST` is a cloud provided email services (e.g. Office 365). | | `NIHMS_MAIL_AUTH` | | The type of authentication. Valid values are: `MS_EXCHANGE_OAUTH2` and `LOGIN`. | | `PASS_DEPOSIT_NIHMS_EMAIL_FROM` | | The official email address that sends the error messages. | | `TEST_DATA_POLICY_TITLE` | | The title of the Policy to associate to the Deployment Test Funder of the Test Grant. | | `TEST_DATA_USER_EMAIL` | | The email of the User to set as the PI on the Test Grant. | | `TEST_DATA_SKIP_DEPOSITS` | true | Whether to skip sending the Deployment Test Deposit to the remote repository or not. | | `TEST_DATA_DSPACE_REPO_KEY` | | The repository key of the DSpace repository, if exists, that will be used to delete Deployment Test Deposit Items if made. | ## Repositories Configuration The Repository configuration contains the parameters used for connecting and depositing custodial material into downstream repositories. The format of the configuration file is JSON, defining multiple downstream repositories in a single file. Each repository configuration has a top-level key that is used to identify a particular configuration. Importantly, each top-level key *must* map to a `Repository` resource within the PASS repository. This means that the top-level keys in `repositories.json` are not arbitrary. In fact, the top level key must be: * the value of a `Repository.repositoryKey` field, which is a `Repository` resource in the PASS repository. Deposit Services comes with a default repository configuration, but a production environment should override the default. Defaults are overridden by creating a copy of the default configuration, editing it to suit, and setting `PASS_DEPOSIT_REPOSITORY_CONFIGURATION` to point to the new location. Deposit Services is configured at runtime by a configuration file referenced by the system property `pass.deposit.repository.configuration` or the environment variable `PASS_DEPOSIT_REPOSITORY_CONFIGURATION`. By default, if the system property or environment variable are not present, the classpath resource `/repositories.json` is used. > Acceptable values for `PASS_DEPOSIT_REPOSITORY_CONFIGURATION` must be a form of [Spring Resource URI](https://docs.spring.io/spring-framework/reference/core/resources.html#resources-implementations). A possible repository configuration is replicated below: ```json { "JScholarship": { "assembler": { "specification": "DSpace", "beanName": "DSpaceAssembler" }, "transport-config": { "protocol-binding": { "protocol": "DSpace" } } }, "PubMed Central": { "assembler": { "specification": "nihms-native-2017-07" }, "transport-config": { "protocol-binding": { "protocol": "sftp", "username": "${pmc.ftp.user}", "password": "${pmc.ftp.password}", "server-fqdn": "${pmc.ftp.host}", "server-port": "${pmc.ftp.port}", "default-directory": "/upload/%s" } } } } ``` ### Customizing Repository Configuration Elements Values may be parameterized by any property or environment variable. To create your own configuration, copy and paste the default configuration into an empty file and modify the JSON as described above. The configuration *must* be referenced by the `pass.deposit.repository.configuration` property, or is environment equivalent `PASS_DEPOSIT_REPOSITORY_CONFIGURATION`. Allowed values are any [Spring Resource path](https://docs.spring.io/spring-framework/reference/core/resources.html#resources-implementations) ( e.g. `classpath:/`, `classpath*:`, `file:`, `http://`, `https://`). For example, if your configuration is stored as a file in `/etc/deposit-services.json`, then you would set the environment variable `PASS_DEPOSIT_REPOSITORY_CONFIGURATION=file:/etc/deposit-services.json` prior to starting Deposit Services. Likewise, if you kept the configuration accessible at a URL, you could use `PASS_DEPOSIT_REPOSITORY_CONFIGURATION=http://example.org/deposit-services.json`. ## Configuring the NIHMS Email Service The NIHMS email service is responsible for parsing emails from NIHMS that contain different error codes on deposits. Currently, NIHMS only communicates its deposit status via email. Setting up an email account that will receive emails from NIHMS is a prerequisite. The email account that is set up should be represented by the value of `NIHMS_MAIL_USERNAME`. There are two authentication protocols supported by this service. A regular SMTP login credentials and Microsoft Exchange OAuth2. By providing a value of `LOGIN` or `MS_EXCHANGE_OAUTH2` for the environmental value, it will determine which type of authentication to use. If using the `MS_EXCHANGE_OAUTH2` authentication type, then `NIHMS_MAIL_TENANT_ID`, `NIHMS_MAIL_CLIENT_ID`, and `NIHMS_MAIL_CLIENT_SECRET` are required as well. ## Configuring the Deployment Test Data Service The Deployment Test Data Service is responsible for cleaning up Submission/Deposit data created by Deployment Tests that run against live PASS environments. To enable the job that executes the Deployment Test Data Service, `pass.test.data.job.enabled` needs to be set to `true`. The `TEST_DATA*` environment variables need to be configured as well. By default, Submissions created by Deployment Tests that are received by Deposit Services are fully processed; however, the Deposit is not actually sent to the remote repository. This can be changed by setting `TEST_DATA_SKIP_DEPOSITS` to `false`, which will then send the Deposit to the remote repository. Additionally, when `TEST_DATA_SKIP_DEPOSITS` is `false`, if the remote repository is a DSpace repository, the Deposit Item in DSpace will be deleted by the Deployment Test Data Service. # Next Steps / Institution Configuration It is important to understand that the Deposit Service is not overly prescriptive. When PASS was developed, we did not\ have a set of requirements for supporting a wide range of repositories, but we acknowledged that possibility existed. So\ a balance was struck with the Deposit Service APIs: if they were over-specified, there may be use cases or repositories\ that couldn't be accommodated later. If they were under-specified, then Deposit Services wouldn't be able to provide\ value by sharing implementations or behaviors between repositories. Since an exhaustive analysis of repository interfaces was impractical at the time, the idea was to provide general\ interfaces that could accommodate almost any scenario. As more use cases, patterns, or abstractions were revealed (by\ onboarding new institutions) they could be formalized (e.g. as Java interfaces). Therefore, supporting new repositories or use cases requires novel code to be developed. ## Supporting a New Repository ### High Level Requirements In order to support a new downstream repository (e.g. Dataverse, Islandora), there are four high-level requirements to\ negotiate: 1. Repository protocol: the protocol used to transmit the bytes to the repository (e.g., SFTP, HTTP). 2. Custody transfer: mechanism used to determine whether a successful custody transfer took place. 3. Package spec: governs physical characteristics of the package (structure, pathing, naming), and required or optional\ metadata (e.g., NIH bulk package spec, BagIT). 4. Metadata mapping: maps elements of the PASS model to package metadata elements. The philosophy of the Deposit Service is that these requirements are a negotiation. This is for a few reasons: * PASS does not want to prescribe or dictate to a downstream repository what it must accept. * Network effects are not in PASS's favor. A popular repository like DSpace is unlikely to change its package\ specifications to accommodate limitations in PASS. * Requirements to accommodate limitations in PASS. In order to satisfy their workflows, institutions will place requirements on packages, especially the metadata contained\ in the package. Universities, the NIH, or the NSF are not going to change their package requirements to conform with\ PASS. ### Rationale for Packages The package-oriented nature of the Deposit Service is the preferred way to send the files for a deposit. Sending a set\ of files in a single archive is the most predictable transport approach for completing the transfer of custody. *It is also possible to send each file separately if the integration requires it, but the DS transport implementation*\ *must handle failure cases.* For example, if there is a failure in the middle of sending a set of file to the repository,\ when DS retries the failed Deposit, the transport needs to ensure the Deposit in the repository is in a "good state"\ before starting the transfer again. ### Interface Overview If implementing a new repository the following interfaces will need to be implemented: * Implementation of Assembler API * Implementation of PackageProvider API * Implementation of Transport API ### Downstream Requirements If the downstream repository integration is receiving a single package, it must: * Unpack the package, * interpret its content, * map it to the native repository model, * and store the bytes of each file. Ideally a downstream repository will have some mechanism to determine and expose the status of custody transfer (i.e. a\ package has been accepted or rejected), but that is not a strict requirement. PASS will still function even if the\ downstream repo doesn't expose the status of a deposit attempt. If the downstream repository integration is receiving files separately, it is up to the DS Transport implementation to\ make the appropriate calls as the repository API specifies. ### Repository Protocol The repository protocol is primarily implemented by the Deposit Service Transport API. The Transport API is responsible\ for using a given protocol to connect to the downstream repository, initiate a transfer of bytes (the package), and\ interpret the response for success or failure. In some cases, the response carries valuable information that must be\ persisted. The transport implementation is also responsible for indicating the location of the package, e.g. the folder (SFTP) or\ collection the package will be transferred to within the downstream repository. These transport hints are\ supplied as key-value pairs to the underlying implementation. If the repository protocol is DSpace API, SFTP, or InvenioRDM the existing implementation may be reused, otherwise write\ your own. If the repository has a use case that is not handled by an existing implementation, it might be accommodated by\ supporting new transport hints for an existing implementation rather than writing a novel implementation. Supporting an\ additional transport hint involves: * Adding and documenting the hint in the appropriate class. * New logic to implement hint behavior. * Writing tests for the new behavior. ### Custody Transfer Determining the status of custody transfer depends on the transport. For DSpace API and InvenioRDM transports, if the\ deposit logic completes successfully, the custody transfer is considered completed. For NIHMS transport, the NIHMS\ repository workflow is a long workflow taking several days to complete. The `NihmsReceiveMailService` reads emails\ from NIHMS with status updates for deposits made to NIHMS and updates the associated Deposit accordingly.`NihmsReceiveMailService` is a job that runs on a periodic basis. In the future, if the DSpace API or InvenioRDM transports require Deposit status updates after the initial deposit, it\ is recommended to use a factory pattern to implement the needed logic to get the deposit status from the downstream\ repository. This logic could be added to the `DepositUpdater` that is already invoked from a job to retry failed\ Deposits. If the downstream repository rejects custody of the package, the only recourse is for the end user to perform another\ submission that addresses the reason(s) for the rejection. The reasons for rejecting a package may not be communicated. In practice, a faculty member would need to: 1. understand that their submission has been rejected 2. know who to reach out to, e.g. a support interface or phone number 3. work with PASS support staff to initiate a new submission that addresses the reasons for rejection PASS support staff would need to comb through logs, contact the downstream repository administrator, or take other\ action to determine the cause of the rejection. PASS has no support for remediating submissions that failed because\ the downstream repository workflow rejected it. ### Package Specification For implementations of the DS `Assembler` and `Package Provider API`, if the packaging spec is NIHMS or BagIT, reuse\ existing implementations. If the packaging spec is DSpace METS, reuse existing implementation, and provide a metadata\ mapping in a package provider. If a new specification is being supported, write a new implementation of the `Assembler`,\ using shared implementations like `AbstractAssembler` and `ArchivingPackageStream`. Concrete subclasses of `AbstractAssembler` are responsible for implementing the `PackageProvider` interface. The complexity of creating a stream is encapsulated in `ArchivingPackageStream`, which is shared across all Assemblers\ that produce a single archive file. They allow for the generation of packages for BagIt, DSpace METS, NIHMS, along with\ simple package formats used for integration tests. #### Important Concepts in the Assembler API: * Custodial content: content supplied by the end user in a PASS Submission; these are PASS File entities. * Supplemental files: not to be confused with the PASS File role option of the same name; these are files that are\ generated by the Assembler, usually to comply with a package specification. E.g. a manifest of files and their\ checksums, or a file containing metadata for the package. End users do not upload supplemental files. They are\ supplied by the Assembler implementation. * Package Provider (discussed in the next topic): responsible for producing the supplemental files included in a package * Packages are streams, and are designed to be relatively performant. For example, a client of the Assembler API can\ open a PackageStream and begin reading from it right away, and the stream won't block as long as the implementation\ continues to supply bytes. #### Important Abstractions in the Assembler API * `Resource` and `ResourceBuilder`: A Resource carries metadata about an individual file in a package: its size, media type,\ file name, and checksums. The ResourceBuilder allows a Resource to be built as the PackageStream is written, and\ different components may contribute to a Resource during this process. For example, one component determines the\ media type of `Resource` and must be invoked at the beginning of streaming the resource (typically the first\ 512 bytes are used). Another component may be invoked after streaming the resource to calculate its checksum. The\ ResourceBuilder allows different components to contribute to the Resource state without sharing the concrete\ implementation. * `MetadataBuilder` and `Metadata`: Metadata and MetadataBuilder are similar to Resource and ResourceBuilder, except\ that they provide for the package metadata, as opposed to individual files within the package. * `StreamWriter` and `DefaultStreamWriterImpl`: Uses the factory pattern to instantiate a new `ResourceBuilder` for each\ resource being streamed. The `DefaultStreamWriterImpl` is responsible for handling the process of writing and packaging\ the files associated with a `DepositSubmission`, and is one implementation of the `StreamWriter`. If you need to support\ a new packaging format that `DefaultStreamWriterImpl` doesn't handle (e.g., a different type of archive format or a\ custom packaging scheme), creating a new class that implements `StreamWriter` would be a preferred method to add\ different behavior to streaming resources. #### Metadata Mapping Implementation of the Deposit Service Package Provider API. Specifically the generation of Supplemental Resources;\ metadata files that are generated by the Deposit Service, not submitted by the end user. Examples of supplemental\ resources include: * BagIt * bagit.txt: a required file in BagIt packages identifying its version and encoding * manifests: a required file in BagIt packages listing the contents and their checksum * bag-info.txt: an optional file in BagIt (though often required by institutional profiles) that provides descriptive\ metadata about the package; this is the volatile section of BagIt # Notification Services Notification Services (NS) is a service component in PASS that provides timely notification (e.g. via email) of events that occur as a Submission moves through a workflow. Notifications may be strictly informational ("a submission was canceled", "a submission was approved"), or they may prompt for action ("please review and submit", "please correct these things"). Notifications are directed to the actors participating in the submission process. NS is a backend service component in PASS written in Java/Spring Boot. NS reacts asynchronously to `SubmissionEvent` messages emitted by the Pass-Core component by composing and dispatching notifications in the form of emails to the participants related to the event. NS is designed to be email-agnostic. For example, implementations could deliver notifications via Slack messages or interactively through reactive JavaScript. The only notification dispatch **currently** implemented is email. ## Proxy Submission Use Case This is currently the use case which Notification Services supports. Here are the high level use case steps: 1. A preparer user creates a Submission. 2. The authorized submitter user receives a notification that a Submission awaits their approval. 3. The authorized submitter user performs the submission, or requests changes from the preparer. 4. If the latter, the preparer updates the Submission, and the authorized submitter is notified. 5. Supports edge cases, such as when the authorized submitter has never logged into PASS (i.e. the person who is going to click submit doesn't have a PASS User). Note that NS is able to support many notification use cases, but the proxy submission notification is the only one currently implemented. This guide steps through various topics on Notification Services: * [Knowledge Needed / Skills Inventory](/developer-documentation/notification-service/ns-know-need) * [Technologies Utilized](/developer-documentation/notification-service/ns-tech-util) * Technical Deep Dive * [Model](/developer-documentation/notification-service/ns-model) * [Business Logic](/developer-documentation/notification-service/ns-business) * [Template](/developer-documentation/notification-service/ns-templates) * [Dispatch](/developer-documentation/notification-service/ns-dispatch) * [Configuration](/developer-documentation/notification-service/ns-configuration) * [Institution Configuration](/developer-documentation/notification-service/ns-new-institution) # Knowledge Needed / Skills Inventory The knowledge needed to understand the NS documentation is a general understanding of the PASS submission workflow and proxy submitter functionality. Understanding the major components and communication channels between the components is also valuable for this document. From a technical perspective the following are recommended: * [Java 17+](https://www.oracle.com/java/technologies/downloads/) * [Spring Boot](https://spring.io/projects/spring-boot) * Messaging and Templating * A general understanding of Service components from a Systems Design perspective # Technologies Utilized The code for NS can be found in the [pass-support repository](https://github.com/eclipse-pass/pass-support/tree/main/pass-notification-service) NS is a backend service that is written in [Java](https://www.java.com/en/) and [Spring Boot](https://spring.io/projects/spring-boot). The NS project is a [Maven](https://maven.apache.org/) project which is used to execute the standard lifecycle tasks for software development (i.e. build/test/package/release) the NS service. The Maven POM is a child of the [`eclipse-pass/pass-support` POM](https://github.com/eclipse-pass/pass-support) which is a child of the [`eclipse-pass/main` POM](https://github.com/eclipse-pass/main). The following is a list of the most significant technologies used in NS: * [Spring Boot](https://spring.io/projects/spring-boot) is used for functionality such as listening for JMS messages, email dispatch, configuration, “wiring-up” of NS components via dependency injection, and tests. * There are unit and integration tests in NS. Tests are executed using [JUnit](https://junit.org/junit5/) and Spring Boot Test. [TestContainers](https://testcontainers.com/) are used for integration tests with pass-core. * [Docker](https://www.docker.com/) is used for building an NS docker image that is used for deployment. # Model
PASS Notification Model

PASS Notification Model

The `Notification` class is the central object in the model. It is used to capture the type of notification for a given submission and to provide the data to the notification template. Let's review the important attributes of `Notification`: `Notification.resouceId`: The PASS entity ID for the `Notification`. `Notification.eventId`: The SubmissionEvent ID for the `Notification`. The SubmissionEvent is the object that captures the submission workflow event, which generates the `Notification`. The table below describes a submission workflow action, the submission event type created, and who receives the notification.
Submission State Diagram

Submission State

| What happened to the Submission | SubmissionEvent Type | Notification Recipient List | | ------------------------------- | ------------------------------------------------- | --------------------------- | | AS Cancelled | CANCELLED | Preparer | | AS Submitted | SUBMITTED | Preparer | | AS Request Changes | CHANGES\_REQUESTED | Preparer | | Preparer Cancelled | CANCELLED | AS | | Preparer Request Approval | APPROVAL\_REQUESTED, APPROVAL\_REQUESTED\_NEWUSER | AS | * AS = Authorized Submitter * Preparer = Proxy who has prepared the submitter on behalf of the AS `Notification.type`: The type of the `Notification`. In this case, there is a 1:1 correspondence between the `SubmissionEvent` type and the `Notification` type. | Notification Type | Description | | --------------------------------- | ------------------------------------------------------------------------------------------------------------ | | SUBMISSION\_APPROVAL\_REQUESTED | Preparer has requested approval of a Submission by an Authorized Submitter | | SUBMISSION\_APPROVAL\_INVITE | Preparer has requested approval of a Submission by an Authorized Submitter who does not have a User in PASS. | | SUBMISSION\_CHANGES\_REQUESTED | Authorized Submitter has requested changes to the submission by the Preparer. | | SUBMISSION\_SUBMISSION\_SUCCESS | Submission was successfully submitted by the Authorized Submitter | | SUBMISSION\_SUBMISSION\_CANCELLED | Submission was cancelled by either the Authorized Submitter or Preparer | `Notification.parameters`: The model that is injected into the templating engine. The `parameters` map carries simple strings or serialized JSON structures. * `TO`, `CC`, `BCC`, `FROM`, and `SUBJECT` are all simple strings. * `RESOURCE_METADATA`, `EVENT_METADATA`, and `LINKS` all contain serialized JSON structures. * Handlebars, the Mustache-based template engine, can navigate the JSON structures to pull out the desired information for email templates. Templates are parameterized by the NS model. This allows for simple variable substitution when rendering the content of an email notification. The template language supported by NS is [Mustache](https://mustache.github.io/), specifically the [Handlebars](https://github.com/jknack/handlebars.java) Java implementation. For details on Mustache and Handlebars, check out the [Handlebars blog](http://jknack.github.io/handlebars.java/) and the [Mustache(5) man page](http://mustache.github.io/mustache.5.html). Sample templates are available in the `templates/` folder of the[`notification-services`](https://github.com/eclipse-pass/pass-docker) container in [pass-docker](https://github.com/eclipse-pass/pass-docker) or in the `HandlebarsParameterizerTest` class. The model provided for template parameterization will depend on the version of NS used, because NS composes and parameterizes the model at compile time. Initially NS provides the following model to the Handlebars templating engine: * `to`: a string containing the email address of the recipient of the notification. * `cc`: a string containing comma delimited email addresses of any carbon copy recipients. * `from`: a string containing the email address of the sender of the notification. * `resource_metadata`: a JSON object containing metadata about the `Submission`: * `title`: the title of the `Submission`. * `journal-title`: the name of the journal that the author accepted manuscript is being published to. * `volume`: the volume of the journal that the author accepted manuscript is being published to. * `issue`: the issue of the journal that the author accepted manuscript is being published to. * `abstract`: the abstract of the `Submission`. * `doi`: the DOI assigned by the publisher to the author accepted manuscript. * `publisher`: the name of the publisher. * `authors`: a JSON array of author objects. * `event_metadata`: a JSON object containing metadata about the `SubmissionEvent`: * `id`: the identifier of the event, a URI to the `SubmissionEvent` resource. * `comment`: the comment provided by the preparer or authorized submitter associated with the `SubmissionEvent`. * `performedDate`: the DateTime the action precipitating the event was performed. * `performedBy`: the URI of the `User` resource responsible for precipitating the event. * `performerRole`: the role the `performedBy` user held at the time the event was precipitated. * `link_metadata`: a JSON array of link objects associated with the `SubmissionEvent`. * Each link object has an `href` attribute containing the URL, and a `rel` attribute describing its relationship to the `SubmissionEvent`. * Supported `rel` values are: * `submission-view`: a link to view the `Submission` resource in the Ember User Interface. * `submission-review`: a link to review and approve a `Submission` in the Ember User Interface. * `submission-review-invite`: a link which invites the recipient of the notification to the Ember User Interface, and subsequently presents the review and approve workflow in the Ember User Interface. # Business Logic Notification Services listens for events during the submission process and generates email notifications to relevant users. The main components of the business logic are `SubmissionEventListener`, `NotificationService`, and `Composer`. The business logic handles events, composes notifications, and includes deep links that guide users directly to the necessary actions within PASS. It ensures that notifications are accurate, secure, and properly validated to maintain workflow integrity. ## SubmissionEventListener The `SubmissionEventListener` is the component responsible for connecting to a messaging source and reading SubmissionEventMessage messages. Note the use of the Spring annotation `JmsListener` on the `processMessage` message. NS uses Spring for messaging configuration, which is managed by the `JmsConfig` class. The NS responds asynchronously to modification of `SubmissionEvent` resources. A `SubmissionEvent` represents the state of the `Submission` in a workflow. Each time a significant step in the submission workflow is taken, a `SubmissionEvent` resource will be created. Notification Services do not create `SubmissionEvent` resources; `SubmissionEvent` resources are created by Ember via Pass-Core as the user moves through the submission workflow. Pass-Core will emit a `SubmissionEventMessage` message indicating the creation of the `SubmissionEvent`, and the Notification Service will listen for, and respond to these messages. ## NotificationService The `NotificationService` is the primary service that contains the business logic associated with reading the `Submission` for the `SubmissionEventMessage` and composing a `Notification` and handing it off for dispatch. If future requirements dictate multiple `Notification`s were to arise from a single `SubmissionEvent`, `NotificationService` would be the starting point for implementing the distribution. ## Composer The `Composer` class does the heavy lifting within the `NotificationService`. It is responsible for composing a `Notification` from the `Submission` and `SubmissionEvent`, including: * Determining the type of `Notification`. * Creating and populating the data structures used in the parameters map. * Invoking the LinkValidator to ensure the `SubmissionEvent` contains valid links for the `Notification` type. * Determining the recipients of the `Notification`, and from whom the `Notification` should come from. After a Notification has been created and populated, it is sent to the `DispatchService`. ### Notification Links Notifications which prompt the recipient to perform an action are to contain a "deep link", that when clicked/activated, takes the user directly to the web page in PASS that requires their attention. For example, a "please review" deep link would resolve directly to the submission overview, with a button allowing the user to approve the submission, cancel the submission, or request further changes. A notification that requests corrections should deep link to a page that highlights the edits or actions required by the user, displaying the comments or instructions from the reviewer. #### Link Replay Because deep links are durably stored (i.e. they can be found in email and used at any time after receipt), they may be replayed after the action has been taken. For example, a user activates a "please review" notification link after they have already performed the review. The user interface or link resolution components will respond in an appropriate way when this occurs. #### Link Security If a user's permissions change between the time a link was generated and the time a link was activated - for example, a User was an Authorized Submitter at the time the link was generated, and they were removed from the Grant by the time the link was activated, the user interface or link resolution components will respond in an appropriate way ensuring the user cannot approve something they don't have permission for. #### User Token Link According to the proxy submitter use case, a preparer may request an authorized submitter review a submission without that submitter existing within PASS. In this case, a `SubmissionEventMessage` with an associated `SubmissionEvent` of type`APPROVAL_REQUESTED_NEWUSER` will be created by Pass-Core and sent to the message queue. The `SubmissionEventMessage` object will have the User Token Link set in the `SubmissionEventMessage.userApprovalLink` attribute. This is the link that is included in the notification for the recipient to click on so a User object is created for them to authorize the submission. #### Link Validation The class `LinkValidator` will use `link-validators` in the notification configuration to validate that the correct URL is present in the `Link` object for the `Notification` type. The `Link` objects are created in the `SubmissionLinkAnalyzer` and then sent to `LinkValidator`. The `link-validators` field in the notification configuration file can be an array of `Notification` types or `*`, which will be a link validation rule that applies to all. # Template Templates are used to customize the subject, body, and footer of email messages that result from a notification. Each notification type has a corresponding template, and the templates and their content are configured in the `notification.json` configuration file. A sample portion of the configuration is below: ```json { "templates": [ { "notification": "SUBMISSION_APPROVAL_INVITE", "templates": { "SUBJECT": "Approval Invite Subject", "BODY": "Approval Invite Body", "FOOTER": "Approval Invite Footer" } }, { "notification": "SUBMISSION_APPROVAL_REQUESTED", "templates": { "SUBJECT": "Approval Requested Subject", "BODY": "Approval Requested Body", "FOOTER": "Approval Requested Footer" } }, { "notification": "SUBMISSION_CHANGES_REQUESTED", "templates": { "SUBJECT": "Changes Requested Subject", "BODY": "Changes Requested Body", "FOOTER": "Changes Requested Footer" } }, { "notification": "SUBMISSION_SUBMISSION_SUBMITTED", "templates": { "SUBJECT": "Submission Submitted Subject", "BODY": "Submission Submitted Body", "FOOTER": "Submission Submitted Footer" } }, { "notification": "SUBMISSION_SUBMISSION_CANCELLED", "templates": { "SUBJECT": "Submission Cancelled Subject", "BODY": "Submission Cancelled Body", "FOOTER": "Submission Cancelled Footer" } } ] } ``` You can see that there is an object identifying each notification type, and for each notification type a `SUBJECT`, `BODY`, and `FOOTER` template may be defined. The value associated with `SUBJECT`, `BODY`, and `FOOTER` may be inline content as seen in the example, or it can be a reference to a Spring Resource URI, e.g.: ```json { "notification": "SUBMISSION_APPROVAL_INVITE", "templates": { "SUBJECT": "classpath:/templates/submission-approval-subject.txt", "BODY": "classpath:/templates/submission-approval-body.txt", "FOOTER": "classpath:/templates/submission-approval-footer.txt" } } ``` Using Spring Resource URIs to refer to the template location is a more flexible and maintainable way of managing notification templates. It allows the templates to be updated in place without having to edit the primary configuration file (`notification.json`) any time a template needs updating. Using Spring Resource URIs also allows template content to be shared across notification types. For example, each notification type could use the same`FOOTER` content. The `CompositeResolver` is responsible for determining whether the value represents inline content, or if it represents a Spring Resource URI to be resolved. Notification Services supports Mustache templates, specifically implemented using Handlebars. Each template is injected with the `parameters` map from the `Notification`. See above for the documented fields of the `parameters` map. It is beyond the scope of this README to provide guidance on using Mustache or Handlebars, but there are some examples in `pass-docker`, and in the `HandlebarsParameterizerTest`. Both inline template content and referenced template content (i.e. Spring Resource URIs) can be Mustache templates. # Dispatch The Dispatch portion of Notification Services is not concerned with populating the Notification; it expects that business logic to have been performed earlier in the call stack. Dispatch does have to adapt a Notification to the underlying transport used, in this case email. This means resolving User to recipient email addresses, and invoking the templating engine for composing email subject, body, and footer. Each `SubmissionEventMessage` received by NS results in the creation of a single Notification, which triggers the dispatch of a single email. Multiple recipients (e.g. using CC or BCC email headers) can be specified on the email if needed. While email is the natural form of dispatching notifications, the model tries to remain independent of an underlying transport or dispatch mechanism. ## Email Implementation The only `DispatchService` implementation is the `EmailDispatchImpl`, which is composed of three main classes: * `Parameterizer`: responsible for resolving template content and invoking the templating engine, producing the content for the email subject, body, and footer. * `EmailComposer`: responsible for adapting the Notification to an email (including resolving and setting the from, to, and cc addresses), provided the parameterized templates. * `JavaMailSender`: responsible for actually sending the email to recipients. ## Composition The `EmailComposer` is responsible for adapting the `Notification` to an email. This includes: * Resolving Notification recipient URIs to email addresses. * In the case of mailto URIs, the scheme specific part is used as the recipient. * In the case of `User` object, the `User.email` is used. * Applying the email recipient whitelist. * Creating the email itself, including the email subject and message body, and encoding. * The subject and message body are provided to the `EmailComposer` by the templating engine. After the `EmailComposer` has created an email, it is returned to the `EmailDispatchImpl` for dispatch via SMTP. # Configuration The required runtime configuration for Notification Services is composed of a configuration file and a set of environment variables. ## Configuration File The NS configuration file is referenced by the environment variable `PASS_NOTIFICATION_CONFIGURATION` or a system property named `pass.notification.configuration`. The value of this environment variable must be a Spring Resource URI, beginning with classpath:/, file:/, or http\://. Here is a sample NS configuration file: ```json { "mode": "${pass.notification.mode}", "recipient-config": [ { "mode": "DEMO", "fromAddress": "demo-pass@mail.local.domain", "global_cc": [ "demo@mail.local.domain" ], "whitelist": [ "mailto:emetsger@mail.local.domain" ] } ], "templates": [ { "notification": "SUBMISSION_APPROVAL_INVITE", "templates": { "SUBJECT": "Approval Invite Subject", "BODY": "Approval Invite Body", "FOOTER": "classpath:/templates/footer.hbr" } }, { "notification": "SUBMISSION_APPROVAL_REQUESTED", "templates": { "SUBJECT": "Approval Requested Subject", "BODY": "Approval Requested Body", "FOOTER": "classpath:/templates/footer.hbr" } }, { "notification": "SUBMISSION_CHANGES_REQUESTED", "templates": { "SUBJECT": "Changes Requested Subject", "BODY": "Changes Requested Body", "FOOTER": "classpath:/templates/footer.hbr" } }, { "notification": "SUBMISSION_SUBMISSION_SUBMITTED", "templates": { "SUBJECT": "Submission Submitted Subject", "BODY": "Submission Submitted Body", "FOOTER": "classpath:/templates/footer.hbr" } }, { "notification": "SUBMISSION_SUBMISSION_CANCELLED", "templates": { "SUBJECT": "Submission Cancelled Subject", "BODY": "Submission Cancelled Body", "FOOTER": "classpath:/templates/footer.hbr" } } ], "link-validators": [ { "rels" : [ "submission-view", "submission-review", "submission-review-invite" ], "requiredBaseURI" : "https://example.org", "throwExceptionWhenInvalid": true }, { "rels": ["*"], "requiredBaseURI" : "http", "throwExceptionWhenInvalid": false } ] } ``` ## Mode Notification Services has three runtime modes: * `DISABLED`: No notifications will be composed or emitted. All JMS messages received by NS will be immediately acknowledged and subsequently discarded. * `DEMO`: Allows a whitelist, global carbon copy recipient list, and notification templates to be configured distinct from the `PRODUCTION` mode. Otherwise, exactly the same as `PRODUCTION`. * `PRODUCTION`: Allows a whitelist, global carbon copy recipient list, and notification templates to be configured distinct from the `DEMO` mode. Otherwise, exactly the same as `DEMO`. Configuration elements for both `PRODUCTION` and `DEMO` modes may reside in the same configuration file. There is no need to have separate configuration files for a "demo" and "production" instance of NS. The environment variable `PASS_NOTIFICATION_MODE` (or its system property equivalent `pass.notification.mode`) is used to set the runtime mode. ## Notification Recipients The recipient(s) of a notification (e.g. email) is a function of a `{Submission, SubmissionEvent}` tuple. After the recipient list has been determined, it can be manipulated as discussed below. ### Whitelist Each configuration mode may have an associated whitelist. If the whitelist is empty, *all* recipients for a given notification will receive an email. If the whitelist is *not empty*, the recipients for a given notification will be filtered, and *only* whitelisted recipients will receive the notification. Having a whitelist for the `DEMO` mode is useful to prevent sending test notifications to unintended recipients. Production should use an empty whitelist (i.e. all potential notification recipients are whitelisted). ### Global Carbon Copy Support Each configuration mode may specify one or more "global carbon copy" addresses. These addresses will receive a copy of each email sent by Notification Services (NS). Global carbon copy addresses are automatically whitelisted and do not need to be explicitly added to the whitelist. Blind carbon copy is also supported. Here is an example recipient configuration that specifies a global carbon copy and a global blind carbon copy: ```json { "recipient-config": [ { "mode": "DEMO", "fromAddress": "pass-noreply@jhu.edu", "global_cc": [ "pass-support@jhu.edu" ], "global_bcc": [ "pass-ops@jhu.edu" ] } ] } ``` Multiple email addresses may be specified. ### Example For example, let's say that NS is preparing to send a notification to `user@example.org`. If the runtime mode of NS is `DEMO` , and: * The `DEMO` mode has no (or an empty) whitelist, then `user@example.org` and the global carbon copy address (for the `DEMO` mode) receives the notification. * The `DEMO` mode has a whitelist that does *not* contain `user@example.org`, then only the global carbon copy address receives the notification. * The `DEMO` mode has a whitelist that *does contain* `user@example.org`, then `user@example.org` and the global carbon copy address receives the notification. * The `DEMO` mode has a whitelist that does *not* contain `user@example.org` and there is no global carbon copy address (for the `DEMO` mode), then no notification will be dispatched. If the runtime mode of NS is `PRODUCTION`, and: * The `PRODUCTION` mode has no (or an empty) whitelist, then `user@example.org` and the global carbon copy address (for the `PRODUCTION` mode) receives the notification. * The `PRODUCTION` mode has a whitelist that does *not* contain `user@example.org`, then only the global carbon copy address receives the notification. * The `PRODUCTION` mode has a whitelist that *does contain* `user@example.org`, then `user@example.org` and the global carbon copy address receives the notification. * The `PRODUCTION` mode has a whitelist that does *not* contain `user@example.org` and there is no global carbon copy address (for the `PRODUCTION` mode), then no notification will be dispatched. ## Environment Variables Supported environment variables (system property analogs) and default values are: * `PASS_NOTIFICATION_QUEUE_EVENT_NAME` (`pass.notification.queue.event.name`): `event` * `PASS_NOTIFICATION_MODE` (`pass.notification.mode`): `DEMO` * `PASS_CORE_URL` (`pass.client.url`): `{PASS_CORE_URL}` * `PASS_CORE_USER` (`pass.client.user`): `{PASS_CORE_USER}` * `PASS_CORE_PASSWORD` (`pass.client.password`): `${PASS_CORE_PASSWORD}` * `SPRING_MAIL_HOST` (`spring.mail.host`): `${SPRING_MAIL_HOST}` * `SPRING_MAIL_PORT` (`spring.mail.port`): `${SPRING_MAIL_PORT}` * `SPRING_MAIL_USERNAME` (`spring.mail.user`): `{SPRING_MAIL_USERNAME}` * `SPRING_MAIL_PASSWORD` (`spring.mail.pass`): `{SPRING_MAIL_PASSWORD}` * `SPRING_MAIL_PROTOCOL` (`spring.mail.transport`): `${SPRING_MAIL_PROTOCOL:SMTP}` * `PASS_NOTIFICATION_CONFIGURATION` (`pass.notification.configuration`): `classpath:/notification.json` In order for notification services to connect to AWS SQS queue (the default messaging provider), the following variables must be set as Environment Variables (System Properties (-Dargs)): * `AWS_REGION` (`aws.region`): AWS region id (i.e. `us-east-1`) In order for NS to access to AWS SQS, standard AWS access management needs to be configured on the deployed NS. See [AWS IAM Access Management](https://docs.aws.amazon.com/IAM/latest/UserGuide/access.html) for more information. For testing purposes, you may set AWS access keys to gain access: * `AWS_ACCESS_KEY_ID` (`aws.accessKeyId`): AWS Access Key to account with access to SQS queue * `AWS_SECRET_ACCESS_KEY` (`aws.secretKey`) : AWS Secret Access Key to account with access to SQS queue **NOTE:** The AWS ID and key should only be used for testing, and in production access should be managed through IAM roles. # Next Steps / Institution Configuration This section describes the configuration and setting up the Notification Services for a new institution. The configuration process involves defining institution-specific settings, managing templates, and integrating with an email system. Please note that the examples provided are for demonstration purposes and may require adjustments based on your institution’s environment and external services setup. ## Prerequisites * Email Service Configuration: Gather your institution's email service settings (e.g., SMTP host, port, authentication details) and ensure the email service is fully operational and ready for integration with Notification Services. * Familiarity with Notification Services: Review the [Notification Services Knowledge Needed](/developer-documentation/notification-service/ns-know-need) * AWS SQS Queue: Notification Services require an SQS queue for message handling. For testing purposes, you can use [LocalStack](https://www.localstack.cloud/) to simulate AWS services in a local environment. ## Setup a Simple Configuration This is a simplified version of the configuration. See the [configuration page](/developer-documentation/notification-service/ns-configuration) and [template page](/developer-documentation/notification-service/ns-templates) for a more in-depth explanation of configuring NS. Be sure to replace email addresses with those from your institution. Save this JSON file to a location that can be referenced by a Docker container or a running JAR file. ```json { "recipient-config": [ { "mode": "DEMO", "fromAddress": "no-reply@institution.edu", "global_cc": ["support@institution.edu"], "global_bcc": ["admin@institution.edu"] } ], "templates": [ { "notification": "SUBMISSION_APPROVAL_INVITE", "templates": { "SUBJECT": "Approval Invite Subject", "BODY": "Approval Invite Body", "FOOTER": "classpath:/templates/footer.hbr" } }, { "notification": "SUBMISSION_APPROVAL_REQUESTED", "templates": { "SUBJECT": "Approval Requested Subject", "BODY": "Approval Requested Body", "FOOTER": "classpath:/templates/footer.hbr" } } ] } ``` ## Run Notification Services in Docker or JAR The easiest way to get notification up and running is to get the latest NS image from our GitHub Container Registry. Run the command below and be sure to replace `1.9.0` with the version you wish to run: ```shell docker run --env=PASS_NOTIFICATION_CONFIGURATION=file:/ns-config/ns-config-json.json --env=AWS_REGION=us-east-1 --env=AWS_ACCESS_KEY_ID={YOUR_ID} --env=AWS_SECRET_ACCESS_KEY={YOUR_KEY} --volume=/path-to-ns-config-file --workdir=/app --runtime=runc -d ghcr.io/eclipse-pass/pass-notification-service:1.9.0 ``` The docker env variable `PASS_NOTIFICATION_CONFIGURATION` points to the volume mounted `/path-to-ns-config-file`, which contains the JSON configuration. The `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are your AWS access ID and key. There are other environment variables to initialize that are found on the [configuration page](/developer-documentation/notification-service/ns-configuration#environment-variables) **NOTE:** The AWS ID and key should only be used for testing and in production this should be managed through IAM roles. You can also download the [Pass Support source code](https://github.com/eclipse-pass/pass-support/releases) and build the NS module to get the NS JAR file that can be invoked by running: ```shell java -Dpass.notification.configuration=file:/ns-config/ns-config-json.json -Daws.region=us-east-1 -Daws.accessKeyId={YOUR_ID} -Daws.secretKey={YOUR_KEY} -jar pass-notification-service-exec.jar ``` The arguments `pass.notification.configuration`, `aws.region=us-east-1`, `aws.accessKeyId`, and `aws.secretKey` are the minimum set to run the JAR; however, a fully functional NS will depend on the fulfillment of the [prerequisites](#prerequisites). ## Supporting Other Types of Queue Services The default queue service is [AWS SQS](https://aws.amazon.com/sqs/). NS can be extended to support other types of queues, but would require further development. Modifying the `JMSConfig` to support other JMS providers will enable support for another type of queue service. # PASS Acceptance Testing Acceptance / smoke tests for the [PASS application](https://github.com/eclipse-pass). This repository utilizes [TestCafe](https://testcafe.io/) to define smoke tests that can run against an instance of PASS. ## Technologies Utilized * [NodeJS](https://nodejs.org/en/) version 24+ * [Yarn](https://yarnpkg.com/) v4.x - package manager, similar to NPM ## Defining Concepts ### Acceptance Tests Acceptance tests ensure that software aligns with user needs and requirements. The goal of acceptances tests is to evaluate the compliance of the system. Acceptance tests help to determine whether the software is acceptable for delivery. ### Smoke Tests Smoke tests are preliminary tests that can reveal simple failures. These tests are designed to catch failures quickly. When smoke tests pass successfully the software is then typically tested with more thorough testing that can take a longer amount of time to complete. Smoke testing can also be referred to as confidence testing, sanity testing, or build acceptance testing. ### TestCafe TestCafe is an open-source test runner that is used for end-to-end testing. ## Running Tests There are two methods for running acceptance tests for the PASS project. One method is running tests against an instance of PASS that is running locally using pass-docker. The other method is to run acceptance tests against a deployed version of PASS that is running on another system, for exmple running tests against a production, development, or staging environment. The following commands are used to run the acceptance and smoke tests against different environments. `yarn run test` - runs the acceptance tests against a locally running pass-docker `yarn run testDeployment` - runs the acceptance tests against a deployed PASS system such as stage or prod. Please see the `rundeploymenttest.sh` file for required environment variables. # PASS Docker Developer-focused PASS runtime, which provides the PASS project and all of its dependent services using docker-compose. PASS Docker provides Docker images that can be used for running PASS in different environments including a local test instance and production. ## Technologies Utilized * [Docker](https://www.docker.com/get-started/) * [Docker Compose](https://docs.docker.com/compose/) ## Local Test Environment The `demo` yml file describes an early system meant to demonstrate new technologies and services that are available in PASS. ### Running A Local Test Instance of PASS In order to run a local test instance of the PASS project using Docker Compose you need to specify the correct `yml` file and the correct `env` file. #### Run Without Deposit Services In order to run a local instance ***without*** deposit-service, SFTP, and dspace, you can run the following command: ``` docker compose -f docker-compose.yml -f eclipse-pass.local.yml up -d --no-build --quiet-pull --pull always ``` #### Run With Deposit Services and DSpace In order to run a local instance ***with*** deposit-service, SFTP, dspace, you can run the following command: ``` docker compose -p pass-docker -f docker-compose.yml -f eclipse-pass.local.yml -f docker-compose-deposit.yml -f docker-compose-dspace.yml up -d --no-build --quiet-pull --pull always ``` **Testing DSpace Integration with a Local Test Instance of PASS** Run the following to create a test admin user in dspace: ``` docker compose -p pass-docker -f docker-compose.yml -f eclipse-pass.local.yml -f docker-compose-deposit.yml -f dspace-cli.yml run --rm dspace-cli create-administrator -e test@test.edu -f admin -l user -p admin -c en ``` Run the following to load sample data into dspace: ``` docker compose -p pass-docker -f docker-compose.yml -f eclipse-pass.local.yml -f docker-compose-deposit.yml -f dspace-cli.yml -f dspace-cli.ingest.yml run --rm dspace-cli ``` #### Run With Deposit Services and InvenioRDM In order to run a local instance ***with*** deposit-service, SFTP, InvenioRDM, you can run the following command: Refer to [PASS Docker Testing InvenioRDM](/developer-documentation/pass-docker/invenio-rdm) for instructions managing a local test InvenioRDM instance that will communicate with `pass-docker`. **Note this configuration for deposit services and InvenioRDM is for local testing only and is not intended for production.** First, start the local test InvenioRDM and add the `Access token` to the appropriate `env` file by following the steps highlighted in [PASS Docker Testing InvenioRDM](/developer-documentation/pass-docker/invenio-rdm). After the InvenioRDM service is up and running, run the following commands in the `pass-docker` directory: ```console docker compose -p pass-docker -f docker-compose.yml -f eclipse-pass.local.yml -f docker-compose-deposit.yml -f docker-compose-deposit-invenio-rdm.yml up -d --no-build --quiet-pull --pull always ``` #### InvenioRDM Submission Requirements When creating a Submission in PASS that will deposit into InvenioRDM with this local test configuration, there are a few requirements for Submission input: * Open a browser and go to * Login using the staff1 user * Create a new submission * On the Grants step, select the `invenio-test-awd-num-1` grant * On the Details step: * Enter a Publisher * Enter Publication Date (format: yyyy-mm-dd) * Enter Author in format \, \ ### Stopping a Local Instance In order to stop a local instance, you can run the following command: ``` docker compose -p pass-docker down -v ``` Note the `-v` to remove the volumes, **this is critical** so on subsequent starts, user data is not duplicated. ## Services ### idp This service runs a Shibboleth Identity Provider using an image from [InCommon Trusted Access Platform Library](https://spaces.at.internet2.edu/display/ITAP/InCommon+Trusted+Access+Platform+Release). Configuration files in the image are overridden on startup by using files in `idp/`. This service is intended for testing only. The first time the idp service runs, the required PKI files will be generated by the docker compose service `cert-gen-idp`. #### Environment variables * `IDP_HOST=http://localhost:9080` * `SP_LOGIN=http://localhost:8080/login/saml2/sso/pass` Separately there is a non-container environment variable `IDP_INTERNAL_PORT` which is used to set the internal port on the IDP container to which 9080 maps. The default is 8080. This can be used to make 9080 support https by setting it to 4443 in the docker compose environment. One way to do this is by adding `IDP_INTERNAL_PORT=4443` to the docker compose command. Note that `-e` should not be used because it is for container environment variables. ### ldap This service runs the 389 Directory Server which is a LDAP server. It is used by the IDP as a source of information on users. The users in `ldap/pass.ldif` are loaded on startup.This service is intended for testing only. ### pass-core * [Repository](https://github.com/eclipse-pass/pass-core) * [Package](https://github.com/orgs/eclipse-pass/packages/container/package/pass-core-main) Presents a JSON:API window to the backend from behind the authentication layer. Swagger is not currently implemented and as a result it is unreachable. This service provides data and web APIs to the application. This service supports SAML and HTTP basic authentication. #### Environment variables * `PASS_CORE_BASE_URL=http://localhost:8080` : Used when generating JSON API relationship links. Needs to be absolute and must change to match deployment environment * `PASS_CORE_POSTGRES_PORT=5432` * `PASS_CORE_BACKEND_USER=backend` * `PASS_CORE_BACKEND_PASSWORD=backend` * `PASS_CORE_APP_LOCATION=http://pass-ui:81/app/` : Resource location of pass ui resources * `PASS_CORE_APP_CSP=default-src 'self';` : Content Security Policy header value * `PASS_CORE_IDP_METADATA=http://idp:8080/idp/shibboleth` : Resource location of IDP metadata * `PASS_CORE_SP_ID=https://sp.pass/shibboleth` : Identifier of pass-core as an SP * `PASS_CORE_SP_KEY=file:///path/key` : Resource location of SP key * `PASS_CORE_SP_CERT=file:///path/cert` : Resource location of SP certificate * `PASS_CORE_LOGOUT_SUCCESS=/app/` : Location user is redirected after logout * `PASS_CORE_LOGOUT_DELETE_COOKIES="JSESSIONID /,shib_idp_session /idp"` : Cookies to delete on logout, "name path" separated by commas. * `POSTGRES_USER=postgres` * `POSTGRES_PASSWORD=postgres` * `JDBC_DATABASE_URL=jdbc:postgresql://postgres:5432/pass` * `JDBC_DATABASE_USERNAME=pass` * `JDBC_DATABASE_PASSWORD=moo` * `PASS_CORE_FILE_SERVICE_TYPE=S3` * `PASS_CORE_S3_BUCKET_NAME=passcorefilestest` * `PASS_CORE_S3_ENDPOINT=http://localstack:4566` ### postgres A standard PostgreSQL database server with minimum modifications. This service's only interaction is with the [`pass-core`](https://github.com/eclipse-pass/pass-core) service. ### pass-ui * [Repository](https://github.com/eclipse-pass/pass-ui) * [Package](https://github.com/orgs/eclipse-pass/packages/container/package/pass-ui) User interface for the PASS application. PASS-UI currently does not handle environment variables nicely, as a result environmental variables are baked into images at build time. The environment variables in the local test environment should not need to be adjusted between different deployment environments. #### Environment variables * `PASS_UI_PORT=81` * `PASS_API_NAMESPACE=data` * `PASS_UI_ROOT_URL=/app` * `STATIC_CONFIG_URL=/app/config.json` * `DOI_SERVICE_URL=/doiservice/journal` * `MANUSCRIPT_SERVICE_LOOKUP_URL=/downloadservice/lookup` * `MANUSCRIPT_SERVICE_DOWNLOAD_URL=/downloadservice/download` * `POLICY_SERVICE_POLICY_ENDPOINT=/policyservice/policies` * `POLICY_SERVICE_REPOSITORY_ENDPOINT=/policyservice/repositories` * `SCHEMA_SERVICE_URL=/schemaservice` * `USER_SERVICE_URL=/pass-user-service/whoami` ### loader A lightweight Docker image that performs a `curl` command in order to bootstrap the environment with data from `demo_data.json` ## Running Acceptance Tests * [Repository](https://github.com/eclipse-pass/pass-acceptance-testing) There is a small set of end-to-end smoke tests that can run against this environment for some validation of changes. These tests run automatically for new PRs that are opened against `main`, but they can also be run locally. In order to do this, first clone the repository with the tests. Once you have the repository cloned, wait for the Docker Compose environment to start up and initialize. Once the Docker Composer environment is initialized run the tests directly. Run the following command from within the `pass-acceptance-testing` directory: ```sh yarn # Installs project dependencies yarn run test # Runs tests ``` ## Related Documentation: * [PASS Docker Testing InvenioRDM](/developer-documentation/pass-docker/invenio-rdm) # Testing InvenioRDM An InvenioRDM instance that can be run locally for testing [`pass-docker`](/developer-documentation/pass-docker). **Note**: This is intended for testing a local instance of `pass-docker` and is not meant for Production use. The `pass-docker-invenio-rdm` directory was created following these instructions: [InvenioRDM Installation](https://inveniordm.docs.cern.ch/install/) ## Technologies Utilized * Python v3.8 or greater installed. * The `invenio-cli` python tool installed and available in your PATH: [InvenioRDM CLI Installation](https://inveniordm.docs.cern.ch/install/cli/) ### Running the InvenioRDM Instance Run the following commands in order to start the InvenioRDM instance: ```console ./build.sh ./start.sh ``` The above commands starts by building the application docker image, once the docker image has been built the commands will start the application and its related services (database, Elasticsearch, Redis and RabbitMQ). The build and boot process will take time to complete. The first time the commands run the docker images will need to be downloaded, the inital downloading of docker images will result in the commands taking longer to complete. ### Accessing the InvenioRDM Instance * Visit in your browser * Login using the credentials for the admin user in `invenio-rdm/pass-docker-invenio-rdm/app_data/users.yaml` * Click on the `user menu button` located in the top right corner * Click on `Applications` * Click `New token` in the Personal access tokens * Enter a Name and click `Create` * Copy the Access token that was created * Paste the token value in the `pass-docker/invenio-rdm/pass-docker-invenio-rdm/.eclipse-pass.invenio.local_env` as the value for the `INVENIORDM_API_TOKEN` property **Note**: The server is using a self-signed SSL certificate, so your browser will issue a warning that you will have to by-pass. ### Stopping the InvenioRDM Instance To stop the InvenioRDM instance, run the following commands: ```console ./stop.sh ``` # Release This section outlines the overall process and steps to perform the community release of PASS. A PASS release produces a set of Java artifacts, Docker images, and Documentation. Java artifacts are published on Sonatype Central Portal and Maven Central repositories. Docker images are pushed to [GitHub Container Registry (GHCR)](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry). Source code is tagged and release notes made available. Each release of PASS has its own version which is used by every component. PASS uses `MAJOR.MINOR.PATCH` [semantic versioning](https://semver.org/) approach. The version should be chosen based on those guidelines. ## Release Steps * Assign a Release Manager, the person who will be responsible for the release process. The Release Manager must be a [PASS committer](https://www.eclipse.org/projects/handbook/#roles-cm). * Choose a release version that communicates the magnitude of the change. * Create a GitHub issue in the main repository using the `Release Checklist Issue` template from the [main repository create new issue page](https://github.com/eclipse-pass/main/issues/new/choose). * Follow the steps in the created Release GitHub issue to complete the release. * See [these instructions](#triggering-the-release-all-github-workflow) for running the `Publish: Release All` GitHub Action workflow. ### Triggering the Release All GitHub workflow * **Before running the `Publish: Release All` workflow, the** [**GitHub PAT configuration**](#github-personal-access-token-setup) **is required.** * Navigate to [Publish: Release All](https://github.com/eclipse-pass/main/actions/workflows/pass-complete-release.yml) * Click on the `Run workflow` dropdown button * Confirm the branch is `main` and enter the versions in the `Release version` and `Next dev version` fields * Release version: full release version, e.g. 1.10.0. These versions should be regarded as immutable. These releases for Java projects cannot be updated or deleted. * Next dev version: snapshot or development versions, e.g. 1.11.0-SNAPSHOT (please use all capital letters for the SNAPSHOT suffix). These development versions are intended to be overwritten. * Click the `Run workflow` button * After a few seconds, a new workflow run should appear in the table with a yellow (in-progress) status dot. Clicking on that will allow you to monitor the run's progress by watching logs. It is recommended that you monitor the automation after triggering it to make sure it completes successfully.
Running Release All Workflow

Running Release All Workflow

### GitHub Personal Access Token Setup The `Publish: Release All` GitHub workflows depend on the `JAVA_RELEASE_PAT` secret having permission to access all the eclipse-pass repositories and write packages.\ You have to create a new classic Personal Access Token (PAT) to do the release (GitHub/Settings/Developer Settings/Personal access tokens/Tokens (classic)).\ If you do so, set the expiration to 7 days, check the `repo` and the `write:packages` scope (subscopes under `repo`\ and `write:packages` will be selected too). How to set the secret in the main repository using the gh command line tool [GitHub CLI](https://cli.github.com/): ``` gh auth login gh secret set JAVA_RELEASE_PAT --body --repo eclipse-pass/main ``` ## Alternate Release Procedures The [Publish: Release All](https://github.com/eclipse-pass/main/actions/workflows/pass-complete-release.yml) GitHub Action workflow is the preferred way to complete the PASS release.\ However, it is possible to execute a PASS release with project automations one at a time or manually if needed. This is not recommended unless absolutely necessary since executing the\ release manually introduces the chance of making mistakes. * [Release Projects One At a Time](/developer-documentation/release/release-steps-project-one-at-a-time) * [Manual Release](/developer-documentation/release/release-steps-manual) # Release Projects One At a Time This section provides the details of performing a release one project at a time with automations by the release manager. [**Publish: Release All**](https://github.com/eclipse-pass/main/actions/workflows/pass-complete-release.yml) **GitHub Action workflow is the preferred way to release PASS. Releasing PASS one project at a time should only be done if absolutely required.** ### Java projects The individual Java projects can be released individually. Release these in the order defined here due to\ dependencies. Between each of these releases, you will need to wait for the Java artifacts to appear on Maven Central.\ This will give you enough time to do other release activities, such as releasing non-Java artifacts. The release\ workflows should wait for you, but checking will mitigate any potential issues with the release. 1. [main](https://github.com/eclipse-pass/main) * [Release workflow](https://github.com/eclipse-pass/main/actions/workflows/release.yml) * [Maven central](https://central.sonatype.com/artifact/org.eclipse.pass/eclipse-pass-parent) 2. [pass-core](https://github.com/eclipse-pass/pass-core) * [Release workflow](https://github.com/eclipse-pass/pass-core/actions/workflows/release.yml) * [Maven Central](https://central.sonatype.com/artifact/org.eclipse.pass/pass-core-main) * [Package](https://github.com/eclipse-pass/pass-core/pkgs/container/pass-core-main) 3. [pass-support](https://github.com/eclipse-pass/pass-support) * [Release workflow](https://github.com/eclipse-pass/pass-support/actions/workflows/release.yml) * [Maven Central](https://central.sonatype.com/artifact/org.eclipse.pass/pass-support) * [Notification Service](https://github.com/eclipse-pass/pass-support/pkgs/container/pass-notification-service) * [Grant Loader](https://github.com/orgs/eclipse-pass/packages/container/package/jhu-grant-loader) * [Deposit Services](https://github.com/orgs/eclipse-pass/packages/container/package/deposit-services-core) * [Journal Loader](https://github.com/orgs/eclipse-pass/packages/container/package/pass-journal-loader) * [NIHMS Loader](https://github.com/orgs/eclipse-pass/packages/container/package/pass-nihms-loader) ### Non-Java projects These can be released in any order. You should release these between releasing Java projects, while waiting for artifacts to become available in Maven Central. * [pass-ui](https://github.com/eclipse-pass/pass-ui) * [Release workflow](https://github.com/eclipse-pass/pass-ui/actions/workflows/release.yml) * [Package](https://github.com/eclipse-pass/pass-ui/pkgs/container/pass-ui) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing) * [Release workflow](https://github.com/eclipse-pass/pass-acceptance-testing/actions/workflows/release.yml) ### Other projects This must be released last because it relies on some of the Docker images that will be published during the release process of the preceding projects. 1. [pass-docker](https://github.com/eclipse-pass/pass-docker) * [Release workflow](https://github.com/eclipse-pass/pass-docker/actions/workflows/release.yml) ### GitHub code release You will have to manually create a GitHub release through the GitHub web interface to complete the release of each of these components. This can generally be done any time after the release automation completes successfully, since the release will be made against a tag created by the automation.
Code Release Section of the Repository

Code Release Section of the Repository

* Navigate to the Releases section of the repository. * Click the "Draft new release" button near the top of the page. * Input release title, matching the release version used for the release (e.g. 0.5.0). * Choose the release tag that was created during the automation. * Click on the "Generate release notes" button to generate a list of changes since the last release. ## Triggering a GitHub workflow Take [eclipse-pass/main](https://github.com/eclipse-pass/main) as an example. * Navigate to the `Actions` tab in your target repository. Select the workflow you want to trigger, e.g. the Release workflow in [Publish: manual full release](https://github.com/eclipse-pass/main/actions/workflows/release.yml) * Click on the `Run workflow` button * Input the branch you wish to run the release against and the desired `Release` and `Next dev` versions * Release version: full release version, e.g. 1.10.0. These versions should be regarded as immutable. These releases for Java projects cannot be updated or deleted. * Next dev version: snapshot or development versions, e.g. 1.11.0-SNAPSHOT (please use all capital letters for the SNAPSHOT suffix). These development versions are intended to be overwritten. * After a few seconds, a new workflow run should appear in the table with a yellow (in-progress) status dot. Clicking on that will allow you to monitor the run's progress by watching logs. It is recommended that you monitor the automation after triggering it to make sure it completes successfully before moving on to release the next project. # Manual Release This section provides the details on performing a release one project at a time manually by the release manager. [**Publish: Release All**](https://github.com/eclipse-pass/main/actions/workflows/pass-complete-release.yml) **GitHub Action workflow is the preferred way to release PASS. Releasing PASS manually should only be done if absolutely required.** ## Required Software The following software is required: | Name | Version | | -------------- | ------- | | Java | 17 | | Maven | 3.8.x | | Docker | 20.10.x | | Docker Compose | 2.x | | Git | | ### Sonatype Developers will need a Sonatype Central Portal account to release Java projects.\ Maven must be configured to use the account by modifying your `~/.m2/settings.xml`. To learn more about Sonatype, documentation is available on their [website](https://central.sonatype.org/publish/publish-portal-guide/). Example pom setup: ```xml central YOUR_SONATYPE_USERNAME YOUR_SONATYPE_PASSWORD central true gpg YOUR_GPG_PASSPHRASE ``` ### GitHub Container Registry (GHCR) Developers will need a GitHub account which is a member of the [eclipse-pass](https://github.com/eclipse-pass) organization. ## Release Sequence If a manual release is required, a specific order must be followed. The Java projects must follow a strict sequence, following its dependency hierarchy. Other Javascript based projects can be released in any order. Both the Java and non-Java releases can be done in parallel, as there are no direct code dependencies between them. 1. [`main`](https://github.com/eclipse-pass/main) 2. Java projects 1. [`pass-core`](https://github.com/eclipse-pass/pass-core) 2. [`pass-support`](https://github.com/eclipse-pass/pass-support) 3. Non-Java projects * [`pass-ui`](https://github.com/eclipse-pass/pass-ui) * [`pass-acceptance-testing`](https://github.com/eclipse-pass/pass-acceptance-testing) 4. [`pass-docker`](https://github.com/eclipse-pass/pass-docker) ## Java Release Maven is used to perform many of the release tasks: * Sets versions and builds * Tests * Pushes release artifacts * May also build Docker images The versions of all the Java artifacts are the same for a release. The parent pom in `main` sets the version to be inherited by all its children; therefore this project needs to be released first, as all other projects need to reference it. After this project is released, other projects are released in an order which guarantees that all PASS dependencies for them have already been released. You will need to wait for artifacts to show up in Maven Central before building a module which depends on them. For convenience, we set and export environment variables RELEASE for the release version, and NEXT for the next development version; e.g., `export RELEASE=0.1.0` and `export NEXT=0.2.0-SNAPSHOT`.\ For each of these child projects, we first clone the source from GitHub, and operating on the principal branch (usually `main`). Update the reference to the parent pom and set the release version. ``` mvn versions:update-parent -DparentVersion=$RELEASE mvn versions:set -DnewVersion=$RELEASE ``` After this, we do build and push the artifacts to Sonatype, commit the version change, and tag it: ``` mvn -ntp -P release clean deploy git commit -am "Update version to $RELEASE" git tag $RELEASE ``` Push any created images to GHCR after logging in. Visit the GitHub docs [Working with the Container registry](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry) for more information. ``` docker push IMAGE_NAME:$RELEASE ``` Push commits and tags to GitHub: ``` git push origin git push origin --tags ``` Finally, the new development code needs to be built and pushed to GitHub. Repeat the process above with the dev version, but do not create the tag. At this point, we should have deployed the release to Sonatype (and eventually to Maven Central), pushed a tag to GitHub, and deployed the new development release to Sonatype. In addition, the project may be released on GitHub. This provides a way to add release notes for a particular project. A GitHub release is done manually by uploading artifacts through the UI. The release version and tag should be the same used for Maven Central. Release notes can be automatically generated from commit messages and then customized. See the GitHub [Java Release Workflow](https://github.com/eclipse-pass/main/blob/main/.github/workflows/release.yml) for the details on the exact commands that are run. ### Manual Release Steps for Main, Pass-Core, Pass-Support * Update POM to release version * Commit release version update * Tag release version * Build and deploy to Sonatype * Push any generated Docker images to GHCR * Update POM to dev version * Commit dev version update * Build and deploy to Sonatype * Wait for artifacts in Maven Central * Push any generated Docker images to GHCR * Push commits to GitHub ## JavaScript Projects The following projects can be released by performing the following steps when the release needs to be performed manually. ### PASS-UI Update the version in `package.json` and in `build.sh`, and commit those changes via a PR to the `pass-ui` repo. Build a new docker image from within the `pass-ui` repo by running: ``` sh build.sh ~/pass-docker/.env ``` Note, you might want to ensure `node_modules` are removed first to ensure a clean build. Push that image to GHCR. For example: `docker push ghcr.io/eclipse-pass/pass-ui:` ### PASS-Acceptance Testing All that's required is to tag a new release in the GitHub UI. After pushing the images to GHCR, update the appropriate image lines in `docker-compose.yml` and `pass-docker` with the new version. Open a pull request against `pass-docker` with these updates. Once acceptance-tests successfully run in CI in your `pass-docker` PR, and once you've done some additional manual spot checking while running `pass-docker` locally, go ahead and tag a new release in the GitHub UI for each of the following projects: `pass-ui` and `pass-acceptance-testing`. ## Testing Manual testing can be done using the newly updated pass-docker to run the release locally. Acceptance testing is run automatically on GitHub against pass-docker/main. ## Post Release * Update release notes * Update project documentation ### Update Release Notes 1. Ensure that there is a milestone for the release. 2. Get a list of all issues that are closed and in the `eclipse-pass` project by going to the [main repository issue list](https://github.com/eclipse-pass/main/issues?page=1\&q=is%3Aissue+is%3Aclosed+project%3Aeclipse-pass%2F4). 3. Check that the correct tickets are in the release milestone. 4. Archive the release tickets in the Project by going to the [Kanban Board](https://github.com/orgs/eclipse-pass/projects/4/views/2), scrolling to the Done column, verifying that all tickets in the list have the new version tag, then selecting the ellipsis button and "Archive all cards". 5. Include in the Release Notes a link to the issues resolved by the release, for example [this milestone](https://github.com/eclipse-pass/main/milestone/11?closed=1). # PASS Infrastructure The PASS project's infrastructure documentation provides a comprehensive overview of the practices, deployment guides, and operational strategies that support the development, delivery, and maintenance of the system. The team employs a Continuous Integration and Continuous Delivery (CI/CD) strategy leveraging GitHub Actions which automates our testing and deployment workflows. Currently, the team is focused on reducing manual interventions within the pipeline, moving towards a fully automated continuous deployment model. The deployment guide details how to set up the PASS platform across various environments, including cloud infrastructure on AWS, using components like EC2, ECS, RDS, and S3. It emphasizes the use of infrastructure-as-code tools such as Docker Compose, and details the deployment workflow. It includes a roadmap for the future dev-ops technologies like Kubernetes, ArgoCD/Flux, Prometheus/Grafana. In the operations and production section, the documentation details the architecture and infrastructure management strategies implemented by JHU to monitor, maintain, and optimize the platform. This assists the JHU team in their own documentation of their production environment, but also serves as one example of a production environment for other adopters of PASS. This includes AWS-based deployment models, monitoring, data management, and ensuring consistent operations across environments. **Table of Contents:** 1. [CI/CD](/infrastructure-documenation/ci-cd) 2. [Deployment](/infrastructure-documenation/deployment) 3. [Operations/Production](/infrastructure-documenation/operations-production) # CI/CD ## Defining Concepts: ### CI: Continuous Integration Continuous integration is a devops software development practice where developers regularly merge code changes into a central repository, after which automated builds and tests are run. ### CD: Continuous Delivery Continuous delivery is a software development practice where code changes are automatically prepared for a release. Continuous delivery deploys all code changes to a testing environment after the build stage. After a successful continuous delivery run developers will have a deployment-ready build artifact that has passed through a standardized test process. ### CD: Continuous Deployment Continuous deployment takes continuous delivery a step further. Continuous Deployment takes the deployment-ready build artifact that passed through the standardized test process and automatically deploys the artifact into a production environment. ### Continuous Deployment vs Continuous Delivery The difference between continuous delivery and continuous deployment is the presence of a manual approval to update to production. With continuous deployment, production happens automatically without explicit approval. ### CI/CD vs. Automation CI/CD is automating the build, release, and deployment process All CI/CD utilizes automation but not all automation is CI/CD ## Current Status The CI/CD pipeline for the PASS project utilizes GitHub Actions in order to automate testing and deployment of some environment assets. Testing reliability has been improved significantly. Currenly GitHub Actions for testing are ran automatically when a pull request is open. Automation for deploying assets is manually triggered during the PASS release process. Currently the pipeline has several areas where manual intervention is required. The PASS team is eager to improve the pipeline. The team is working towards improving the CI/CD pipeline to achieve reliable continuous delivery. Once the team is confident in the continuous delivery the team hopes to implement continuous deployment. ### GitHub Actions The PASS project utilizes GitHub Actions in order to build a CI/CD pipeline. The GitHub Actions can be triggered using multiple methods. The trigger methods include: Event Trigger, Manual Trigger, and Scheduled Trigger. An event trigger kick starts a GitHub Action when a certain event occurs, for example when a pull request is opened. A manual trigger kick starts a GitHub Action when someone tells the action to run. A scheduled trigger kick starts a GitHub Action at a scheduled time. ### Pull Request Submission Trigger When a pull request is opened multiple GitHub Actions are triggered automatically. One of the GitHub Actions focuses on continuous integration concepts. The continuous integration GitHub Action runs several forms of tests. The following tests are run: Acceptance Tests, Unit Tests, and Integration Tests. Another GitHub Action is triggered during the opening of a pull request, the Git ECA Validation Status actioon. The Git ECA Validation Status action verifies that the author of the pull request is covered by necessary legal agreements to contribute to an Eclipse Foundation Project. Once the actions have completed the GitHub user interface shows if the tests passed or failed. ### CI/CD Workflow Proposal The team is in the process of implementing the following proposal workflow:
## Related Documentation: * [Release Process](/developer-documentation/release) * [Acceptance Testing](/developer-documentation/pass-acceptance-testing) # Code Quality Analysis PASS uses the code quality and security tool [SonarQube Cloud](https://www.sonarsource.com/products/sonarcloud/) to\ ensure a high-quality code base. SonarSource graciously allows open source projects to use a free tier of SonarQube\ Cloud, which integrates directly with our GitHub repositories. ## Summary SonarQube performs static analysis on the PASS codebase to detect bugs, vulnerabilities, code smells, and security\ hotspots. This integration provides automated code quality checks on pull requests, helping to promote and maintain\ clean code. Analysis is triggered automatically on pull requests and merges to the main branch via GitHub Actions. In\ addition, a [plugin](https://docs.sonarsource.com/sonarqube-for-ide/intellij/) can be added to various IDEs, catching\ code quality issues before submitting a pull request. ### List of Repositories on SonarQube * [PASS Main](https://sonarcloud.io/project/overview?id=eclipse-pass_main) * [PASS Core](https://sonarcloud.io/project/overview?id=eclipse-pass_pass-core) * [PASS Support](https://sonarcloud.io/project/overview?id=eclipse-pass_pass-support) * [PASS UI](https://sonarcloud.io/project/overview?id=eclipse-pass_pass-ui) ## Knowledge Needed / Skills Inventory * Understanding code quality concepts * Git/GitHub ## Technologies Utilized * [SonarQube Cloud](https://www.sonarsource.com/products/sonarcloud/): The cloud-based platform hosting the analysis\ engine and results dashboard. * [GitHub Actions](https://docs.github.com/en/actions): publishes results to SonarQube Cloud via workflows. * [SonarScanner for Maven](https://docs.sonarsource.com/sonarqube-cloud/advanced-setup/ci-based-analysis/sonarscanner-for-maven/) * [SonarScanner for Maven GitHub](https://github.com/SonarSource/sonar-scanner-maven) ## Technical Deep Dive ### SonarQube Configuration The full documentation for getting started with SonarQube Cloud is available on their [documentation site](https://docs.sonarsource.com/sonarqube-cloud/getting-started/sign-up/).\ On the pass project it is integrated into our CI/CD pipeline, providing status checks on our pull requests. ### Reading the Reports Access the SonarQube reports using the links in the [Summary](#list-of-repository-on-sonarqube) section. Detailed\ guidance on analyzing reports is available on the[SonarQube Cloud Documentation site.](https://docs.sonarsource.com/sonarqube-cloud/digging-deeper/overview/) Key areas to examine include: * Project Overview (Main Dashboard): * Quality Gate status (Passed/Failed) – this is the primary indicator of code health. * The main **Ratings** (A-E) for Reliability, Security, Maintainability, and the **Coverage** percentage for\ a quick code coverage assessment. * Pull Request Analysis (Viewed in GitHub): * When analysis runs on a pull request, SonarCloud adds a status check to the PR in GitHub. * Issues Tab (in SonarCloud Project): * Provides a detailed, filterable list of all identified issues. * Filter by type (Bug, Vulnerability, Smell, Hotspot), severity (Blocker, Critical, Major, Minor, Info),\ status (Open, Confirmed, False Positive, Won't Fix), assignment, creation date, etc. * Measures Tab (in SonarCloud Project): * Explore metrics in more detail. View graphs showing trends over time for size, complexity, coverage, technical\ debt, and issue counts. * Useful for understanding the overall health trends of the codebase. * Code Tab (in SonarCloud Project): * Browse the source code directly within SonarCloud. * Issues are highlighted inline, making it easy to see problems in context. ### Integration with JaCoCo SonarQube does not provide code coverage out-of-the-box, but it does integrate with coverage tools. In a simple project,\ the [setup](https://docs.sonarsource.com/sonarqube-cloud/enriching/test-coverage/java-test-coverage/)\ is trivial, but with the PASS project there are a few extra configuration steps for proper integration within our CI/CD\ pipeline. These extra steps are detailed on the [JaCoCo page](/infrastructure-documenation/sonar-qube/jacoco) of the code quality analysis. ### Known Limitations using Free Tier Subscription * Can only analyze the `main` branch and pull requests (only if `main` is the target branch) of a repository. * Can only use the default [Sonar Way quality gate](https://docs.sonarsource.com/sonarqube-cloud/standards/managing-quality-gates/)\ for code quality analysis * Maximum number of organization members is 5. The full set of limitations for SonarQube Cloud can be found on their [subscription comparison table](https://docs.sonarsource.com/sonarqube-cloud/administering-sonarcloud/managing-subscription/subscription-plans/). ## Related Information * [SonarQube Cloud Homepage](https://www.sonarsource.com/products/sonarqube/) * [SonarQube Cloud Documentation](https://docs.sonarsource.com/sonarqube-cloud/) * [SonarSource Project for Java](https://github.com/SonarSource/sonar-java) * [SonarSource Scanning Examples](https://github.com/SonarSource/sonar-scanning-examples) # Code Coverage This section details how PASS utilizes JaCoCo for measuring Java code coverage in [PASS Core](https://github.com/eclipse-pass/pass-core)\ and [PASS Support](https://github.com/eclipse-pass/pass-support). Unit and Integration tests exist in PASS UI as well, but\ at the moment do not have any code coverage analysis being performed. We anticipate adding code coverage reports to PASS\ UI in the future. ## Summary PASS employs [JaCoCo (Java Code Coverage Library)](https://www.jacoco.org/jacoco/) to measure the extent to which the\ project's Java code is exercised by unit and integration tests. This process typically runs as part of the standard build cycle managed by our build tool (Maven) within the\ CI/CD pipeline (GitHub Actions). JaCoCo generates reports, which are then consumed by SonarQube Cloud to display\ coverage metrics on the project dashboard and pull request analyses. Tracking code coverage helps ensure that critical\ parts of the application are adequately tested, increasing confidence in code quality and reducing the risk of\ regressions. ## Knowledge Needed / Skills Inventory * Understanding Code Coverage Concepts * Java 17+ * Maven build tool ## Technologies Utilized * [JaCoCo](https://www.jacoco.org/jacoco/) * Build Tool [Maven](https://maven.apache.org/) * [GitHub Actions](https://docs.github.com/en/actions) * [SonarQube Cloud](https://www.sonarsource.com/products/sonarcloud/) * [SonarScanner for Maven](https://docs.sonarsource.com/sonarqube-cloud/advanced-setup/ci-based-analysis/sonarscanner-for-maven/) * [SonarScanner for Maven GitHub](https://github.com/SonarSource/sonar-scanner-maven) ## Technical Deep Dive The PASS project integrates JaCoCo within the CI/CD pipeline. Achieving this involves two main steps: configuring JaCoCo\ reports in Maven and integrating the analysis within GitHub Actions. ## Setting up the POM Both PASS Core and PASS Support are multi-module projects which require a specific setup * Agent Preparation: The JaCoCo Java agent is configured to run before the test execution phase. This is done using the`prepare-agent` goal in the parent `pom.xml`. * Aggregate Report Module: A module is designated as the aggregate report module, which collects all the reports from\ the individual modules in the project. This module's `pom.xml` runs the `report-aggregate` goal during the `verify`\ stage. This module is named: * for PASS Core: `jacoco-aggregate-report-pass-core` * for PASS Support: `jacoco-aggregate-report-pass-support` * Individual modules must declare the `prepare-agent` goal in their `pom.xml` * SonarQube * The SonarQube Scanner is added to the parent `pom.xml` * The following properties are added to the parent `pom.xml`: * `sonar.projectName` * `sonar.projectKey` * `sonar.coverage.jacoco.xmlReportPaths` * Report Generation: The report is saved in `jacoco-aggregate-report-[project-name]/target/site/jacoco.xml` ## GitHub Actions The integration with SonarQube also occurs during the CI/CD phase of the build process. To accomplish this integration\ the following steps were implemented in GitHub Actions: * A `SONAR_TOKEN` secret, obtained from SonarQube Cloud, must be configured in the eclipse-pass GitHub organization\ secrets. * In the `snapshot.yml` and `ci.yml` of `eclipse-pass-parent` the SonarQube Scanner is invoked during the `mvn` `deploy`\ and `verify`. * The `fetch-depth` must have a value of 0, so that the entire Git history is fetched. If this isn't set then incomplete\ analysis will be performed. ## Other Configurations to Note * Using GitHub Actions requires the `Automatic Analysis` to be turned off on the project in SonarQube so that GitHub\ Actions can trigger analysis. * A SonarQube project was created for `eclipse-pass-parent`. With 'Automatic Analysis' disabled in SonarCloud, all scans\ rely on GitHub Actions triggers. Since the primary workflows may focus only on pull request analysis, linking`eclipse-pass-parent` is a necessary part of the configuration that enables analysis results for the `main` branch\ (of Core/Support) to be processed and displayed in SonarQube. Additionally, this allows the `eclipse-pass-parent`\ project itself to be analyzed for code quality and dependency vulnerabilities. ## Related Information * [JaCoCo Official Website](https://www.jacoco.org/jacoco/) * [JaCoCo Maven Plugin Documentation](https://www.jacoco.org/jacoco/trunk/doc/maven.html) * [PASS SonarQube Documentation](/infrastructure-documenation/sonar-qube) * [SonarQube Cloud Code Coverage Integration](https://docs.sonarsource.com/sonarqube-cloud/enriching/test-coverage/java-test-coverage/) # Deployment ## Table of Contents 1. [Summary](#summary) 2. [AWS Infrastructure Components](#aws-infrastructure-components) 3. [Deep Dive: Deployment & Release](#deep-dive-deployment--release) 4. [PASS Deployment Process](#pass-deployment-process) 5. [PASS Release Process](#pass-release-process) 6. [Related Information](#related-information) ## Summary The Public Access Submission System (PASS) is an open-source platform designed to streamline compliance with funder and institutional open access policies. This guide outlines the deployment process for PASS, which is adaptable to various architectures including cloud, hybrid, or on-premises environments. > **Note**: PASS is transitioning towards a cloud-native version. Expect ongoing changes to the architecture, infrastructure, and deployment process, such as moving from Docker Compose to Kubernetes or implementing Infrastructure as Code with Terraform. See (roadmap)\[./roadmap.md] for more information. ## AWS Infrastructure Components The current PASS infrastructure in AWS includes: * **EC2**: Hosts Docker Compose * **ECS**: Hosts auxiliary microservices * **RDS**: Stores metadata * **S3**: Stores binary data (managed by OCFL) * **ALB**: Provides SSL for the frontend * **WAF**: Protects the frontend ## Deep Dive: Deployment & Release ### Prerequisites * Docker and Docker Compose * Git ### PASS Deployment Process 1. Install dependencies: ```bash apt-get -y update apt-get install -y gnupg2 pass docker compose ``` 2. Clone the repository: ```bash mkdir -p /src cd /src git clone git@github.com:eclipse-pass/pass-docker.git cd pass-docker git checkout minimal-assets ``` 3. Run PASS: ```bash cd /src/pass-docker && \ docker compose pull && \ docker compose up ``` ### PASS Release Process PASS uses semantic versioning (`MAJOR.MINOR.PATCH`). The release process includes: 1. Code contribution 2. CI/CD via GitHub Actions 3. Building and testing 4. Generating release artifacts (Java artifacts and Docker images) 5. Publishing artifacts to repositories 6. Triggering deployment via AWS SQS 7. Updating infrastructure with new artifacts 8. Generating release notes #### GitHub Actions Release Workflow The "Publish: Release All" workflow automates the release process: 1. Builds and tests components 2. Publishes Java artifacts 3. Builds and pushes Docker images 4. Creates GitHub Releases For detailed configuration, refer to the [pass-complete-release.yml](https://github.com/eclipse-pass/main/blob/main/.github/workflows/pass-complete-release.yml) Actions workflow. Please refer to [github-cicd.md](/infrastructure-documenation/deployment/github-cicd) for further information on its useage. ## Related Information * [PASS main repository](https://github.com/eclipse-pass/main) * [PASS Docker repository](https://github.com/eclipse-pass/pass-docker) For further assistance or questions, please open an issue in the [PASS main repository](https://github.com/eclipse-pass/main/issues) or find us in the [PASS Slack](https://eclipse-pass.slack.com). # GitHub CI/CD ## Summary GitHub Actions is a powerful [CI/CD](/infrastructure-documenation/deployment/github-cicd) platform integrated directly into GitHub repositories. It allows you to automate various software development workflows, including building, testing, and [deploying](/infrastructure-documenation/deployment) your code. ### Key Concepts 1. **Workflows**: YAML files that define a set of jobs to be executed when triggered by an event. 2. **Jobs**: A set of steps that execute on the same runner. 3. **Steps**: Individual tasks that can run commands or actions. 4. **Actions**: Reusable units of code that can be shared across workflows. 5. **Events**: Specific activities that trigger a workflow run. ### Workflow Action Structure ```yaml name: CI on: [push, pull_request] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Run a script run: echo Hello, world! ``` This simple workflow runs on push and pull request events, checks out the repository, and runs a simple command. ## GitHub Secrets GitHub secrets are encrypted environment variables used to store sensitive information securely. They are crucial for handling authentication and other confidential data in your workflows. ### Types of Secrets 1. **Organization Secrets**: Available to all repositories in the `eclipse-pass` organization. 2. **Repository Secrets**: Specific to a single repository. 3. **Environment Secrets**: Tied to a specific environment within a repository. ### Creating Secrets Due to permission restrictions, PASS project members should use the provided Python script to create repository or environment secrets: ```bash python github_secrets.py -u -t -r -n -v [-e ] ``` For organization secrets, open a ticket with the [Eclipse Help Desk](https://gitlab.eclipse.org/eclipsefdn/helpdesk). ### Using Secrets in Workflows Reference secrets in your workflows like this: ```yaml ${{ secrets.SECRET_NAME }} ``` For reusable workflows, pass secrets explicitly: ```yaml jobs: call-publish-docker: uses: ./.github/workflows/docker-publish.yml secrets: AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} ``` In the called workflow, declare expected secrets: ```yaml on: workflow_call: secrets: AWS_ACCESS_KEY_ID: required: true AWS_SECRET_ACCESS_KEY: required: true ``` ### Best Practices 1. Use reusable workflows for common tasks to maintain DRY principles. 2. Leverage GitHub-hosted runners when possible to reduce maintenance overhead. 3. Use environment protection rules for sensitive deployments. 4. Regularly audit and rotate your secrets. 5. Use GitHub Actions marketplace for pre-built actions to speed up development. ## AWS Integration To interact with AWS services, including ECR (Elastic Container Registry), you'll need to set up appropriate secrets and use AWS-specific actions in your workflows. ### Setting up AWS Credentials Store your AWS credentials as secrets: 1. `AWS_ACCESS_KEY_ID` 2. `AWS_SECRET_ACCESS_KEY` ### Example Workflow for AWS Deployment ```yaml name: Deploy to ECR on: push: branches: [main] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Configure AWS credentials uses: aws-actions/configure-aws-credentials@v1 with: aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} aws-region: us-east-1 - name: Login to Amazon ECR id: login-ecr uses: aws-actions/amazon-ecr-login@v1 - name: Build, tag, and push image to Amazon ECR env: ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }} ECR_REPOSITORY: my-ecr-repo IMAGE_TAG: ${{ github.sha }} run: | docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG . docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG ``` This workflow builds a Docker image and pushes it to Amazon ECR. ## Related Information For more detailed information, refer to the [GitHub Actions documentation](https://docs.github.com/en/actions). # Operations/Production The operations and production workflows, architecture, and infrastructure are composed of a variety of technologies, layers, and techniques. The main important concept and understanding of this guide, is that PASS is designed to be platform independent, with a few exceptions. This guide will describe and summarize the way JHU decided to deploy the application, but this doesn't mean it can't be done another way. One way of deploying PASS is using Amazon Web Services (AWS) cloud computing services. By using the scalable infrastructure we are able to quickly adapt to different demands of usage. By using AWS, it opens up the ability to use infrastructure as code which gives more efficiencies by enabling reuse of infrastructure between environments and aids in the CI/CD pipeline, providing consistent and quick deployments. In addition to deploying PASS, other operation activities will be described such as monitoring, harvesting and loading data, and other communication between different services. If you haven't already, a quick review of the [Welcome Guide PASS Architecture](/welcome-guide/deployment-architecture) article will provide a good foundation for understanding the operations and production environment of PASS. In this guide we step through various topics on JHU Operations and Production: * [Knowledge Needed / Skills Inventory](/infrastructure-documenation/operations-production/ops-know-need) * [Technologies Utilized](/infrastructure-documenation/operations-production/ops-tech-util) * Technical Deep Dive * [PASS Design & Amazon Web Services (AWS) Architecture](/infrastructure-documenation/operations-production/ops-aws-arch) * [PASS AWS Architecture Cost Estimates](/infrastructure-documenation/operations-production/ops-aws-cost) * [Versioning](/infrastructure-documenation/operations-production/ops-version) * [How to Deploy](/infrastructure-documenation/operations-production/ops-deploy) * [Monitoring](/infrastructure-documenation/operations-production/ops-monitor) * [NIHMS Credentials Configuration](https://github.com/eclipse-pass/pass-documentation/blob/main/infrastructure-documenation/operations-production/ops-nihms.md) * [Data Loaders](/infrastructure-documenation/operations-production/ops-loaders) * [Data & Backups](/infrastructure-documenation/operations-production/ops-data-backup) * [Eclipse Operations](/infrastructure-documenation/operations-production/ops-eclipse) * [Next Steps / Institution Configuration](/infrastructure-documenation/operations-production/ops-new-institution) # Knowledge Needed / Skills Inventory Running PASS in a production environment requires a variety set of skills. The main set of knowledge and skills for running PASS are the following: * **AWS Architecture and Development:** knowledge of designing, deploying, and managing applications on AWS. * **Docker and Containers**: knowledge of containerization technologies like Docker, including how to build, deploy, and manage containers in production using ECS or another container orchestration technology. * **Systems Administration**: general knowledge of systems administration is valuable such as managing Linux-based EC2 instances, monitoring and performance tuning, backups, and cost estimations. # Technologies Utilized The PASS production infrastructure and operations are composed of a variety of technologies. The chosen technology stack was desirable for its ease of setup, support for CI/CD, and scalability. ## Technologies Utilized * [Amazon Web Services (AWS) Cloud Computing](https://aws.amazon.com): Many AWS resources are used for the majority of the infrastructure including virtual servers (EC2), Amazon Relational Database service (RDS, using a PostgreSQL instance), Elastic Container Service (ECS), and a variety of others. * [AWS Account Management](https://aws.amazon.com/account/): Used to create Service Roles and "least privilege" Policies that are used by AWS resources running PASS components. * [AWS Command Line Interface](https://aws.amazon.com/cli/): Used in the development, testing, and management of the JHU PASS infrastructure. * [Docker](https://www.docker.com/): Docker is used in all our environments for running the main components of the PASS application. * [GitHub Actions](https://docs.github.com/en/actions): GitHub Actions are used for deployment and CI/CD operations. # PASS Design & AWS Architecture The PASS design supports flexible and scalable submission workflows of research publications. It consists of four main components: PASS UI, Data Loaders, PASS Core, and Deposit Services. PASS Core acts as the central backbone, coordinating the workflow through, Elide, a JSON:API that interfaces with various services. It stores documents using the [Oxford Common File Layout (OCFL)](https://ocfl.io/) on disk or through an S3 bucket, while the backend services are written in Java using Spring Boot. Data Loaders handle the ingestion of journal, grant, and publication data, while Deposit Services manage the deposits of submissions into various repositories like DSpace, NIH, and InvenioRDM. It is important to note that the AWS architecture described in this article reflects the choices made by JHU on how to deploy PASS. The deployment can be done in various ways, including being hosted on other cloud providers or operated on infrastructure managed locally by the institution. The different components of PASS and their associated production setup and operations that will be discussed in this article are specific to JHU. Some of these items may be needed in another PASS deployment, but some may not all depending on the institution's architectural decisions. The JHU PASS architecture is deployed on AWS, leveraging services such as Elastic Compute Cloud (EC2), Elastic Container Service (ECS), Simple Storage Service (S3), and Simple Queue Service (SQS) to ensure high availability, scalability, and security. This setup allows PASS to efficiently handle submission workflows and deposit services across diverse environments, adhering to institutional policies and data governance standards. A quick review of the PASS design and application architecture from our [Deployment and Architecture page](/welcome-guide/deployment-architecture) in the [PASS Welcome Guide](/welcome-guide). Both the PASS Design diagram and Application Architecture diagram are below for a quick reference:
PASS Design Diagram

PASS Design Diagram

PASS Application Diagram

PASS Application Diagram

## JHU PASS AWS Resources Details ### Virtual Private Cloud (VPC) The infrastructure operates across two VPCs. The main VPC, hosts the PASS components and is equipped with both public and private subnets, without direct connectivity to the JHU network. The secondary VPC, includes a private subnet with connectivity to the JHU network, facilitating secure data access and integration with resources within the JHU networks, such as the Grant loader and JHU's institutional repository. ### Identity and Access Management (IAM) and Security Groups AWS IAM and security groups are used to manage the access and security of the PASS application. Following best security practices IAM Roles have been created with "least privilege" policies and assigned to the resources. Additionally, Security Groups have been created and assigned to AWS resources to restrict inbound and outbound connections to align with "least privilege" access. ### EC2 - PASS Core and UI PASS Core and UI are deployed on an EC2 instance. This instance is part of the main VPC and is configured to run the PASS Core API/UI. Configuration files and deployment scripts for PASS Core/UI are stored in S3, allowing for easy access and management during deployments and updates. In addition, the AWS Systems Manager parameter store contains configuration of environment variables in the PASS application. PASS Core uses SQS for message queuing to handle asynchronous communication between different components of the PASS system. For example, when a submission is made, a message is placed in an SQS queue, which is then processed by the deposit services. PASS-UI interacts with AWS infrastructure by utilizing the EC2 instance for hosting, ALBs for managing traffic, and S3 for storing static assets. #### SSO - Shibboleth PASS Core/UI integrates with Shibboleth for authentication, providing a single sign-on experience for users. Shibboleth is a single sign-on system that is used at JHU. The interoperability of this depends on the institution and the header variables in the web browser that are passed by the identity provider. Please contact the PASS team for more information regarding how to integrate with SSO. ### RDS - PASS Database PASS Core uses Amazon RDS, specifically a PostgreSQL database, to store and manage data related to publications, submissions, grants, and policies. The RDS instance resides within the same VPC and is not publicly accessible, consequently this means devops cannot connect directly to the database using SQL tools; however by using a jumpbox it is possible to connect via an SSM managed session if needed. The EC2 instances running PASS Core communicate with the RDS instance over the VPC’s internal network. Security groups are configured to allow traffic between PASS Core and the RDS database, ensuring secure data access. ### Elastic Container Services (ECS) #### Deposit Services Deposit Services are responsible for processing submission data from PASS Core and ensuring it is correctly formatted and packaged for deposit into external repositories. This involves transforming submission metadata and bundling content according to repository requirements. It is deployed as containerized applications using ECS Fargate. As described in the Application Architecture, Deposit Services leverages SQS to handle the asynchronous processing of deposit tasks. When a submission is ready for deposit, PASS Core places a message in an SQS queue, which Deposit Services then pick up for processing. The Deposit Service ECS task is configured to be able to access ECR, SSM Parameter store, SQS, and S3. Deposit services read messages from SQS queues. Deposit Services reads the repository.json configuration file from an S3 bucket. #### Notification Services Notification Services are responsible for consuming messages from the submission event queues and sending notifications via email. The Notification Service ECS task is configured to be able to access the ECR, SSM Parameter store, and SQS. SES is enabled in production mode for sending emails from the notifications service. #### Data Loaders The Data Loaders (Journal, Grant, Publication) are executed as AWS Batch jobs within ECS Fargate compute environments, each designed to accommodate specific data processing needs. The batch jobs leverage AWS services like S3 for configuration management, EventBridge Scheduler for scheduling batch jobs, and ECS Fargate for executing the containerized data loaders. The data loaders have the required permissions for accessing resources like S3 buckets and connecting to the PASS Core API. ### Application Load Balancers (ALB) There are two ALBs, one for handling external traffic and another for internal communication between the different components of PASS. The public load balancers handles requests for the domain name. All traffic is forwarded to HTTPS, and there is a security group that is attached, allowing only inbound/outbound requests to specific ports. The other load balancer for internal communication has as similar setup but is private meaning it is not accessible outside the VPC private network. Similarly, it has an attached security group for permitting specific ports for inbound/outbound requests. #### Target Groups Target Groups are used by ALBs to forward requests to the EC2 instances running the Pass-core/Pass-UI docker containers. There are two target groups, one for the public ALB and one for the private ALB. ### Web Application Firewall (WAF) A WAF sits at the edge of the architecture boundary, as pictured in the PASS Application Architecture diagram, between the end users and the PASS ALB. It is responsible for applying a set of rules to filter out traffic and protect the internal virtual network. ### Certificate Manager The certificate for the PASS public domain name for JHU is in AWS Certificate Manager. ### Simple Email Service (SES) SES is enabled in production mode for sending emails from the notifications service. ### Elastic Container Registry (ECR) ECR is a fully managed Docker container registry that makes it easy for developers to store, manage, and deploy Docker container images. It's leveraged in the PASS Application Architecture by giving a private repository to store the released Docker images for PASS Core, PASS UI, Data Loaders, Deposit Services, and Notification Services. ### SQS Queues SQS publication queue serves as a decoupling mechanism between the PASS Core and Deposit and Notification Services. Submissions are processed asynchronously, resulting in a system that remains responsive and efficient. There are three queues that are used by the overall PASS environment: `deposit`, `submission`, and `submission-event`. ### S3 PASS uses S3 for configuration, deployment, and file storage. PASS Core uses Amazon S3 to store submission-related documents and metadata. The OCFL is employed to organize these files, providing options for local disk storage or S3 bucket retention. The configuration of the File Service in PASS Core is done through environment variables that are set in the Systems Manager parameter store. Public access is blocked on all S3 buckets. # AWS Cost Estimates ## Operations/Production - AWS Cost Estimates In making the decision to use AWS for PASS production infrastructure, cost will invariably be a deciding factor. This article describes the various cost factors and cost estimates sampling of running such an architecture as described in [PASS Design & AWS Architecture](/infrastructure-documenation/operations-production/ops-aws-arch). Costs will of course be variable and dependent on the exact implementation. ### AWS Resource Cost Factors The table below contains the AWS resources used by JHU and the configuration options that will impact the cost for each resource most significantly. The order of the resources is based on percentage of the monthly expense, for instance AWS RDS is the most expensive resource in our implementation. The JHU Configuration column has details for each resource as of September 2024 for what our system currently requires for compute, storage, IO, and monitoring. The [AWS Pricing Calculator](https://calculator.aws/#/) can be used to estimate cost for a PASS application AWS architecture to your specifications. #### AWS Relational Database Service (RDS) * **Cost Factors** * Database Instance Class * Storage Size/IO Bandwidth * Snapshot retention * Data Transfer * Database License * **JHU Configuration** * AWS RDS PostgreSQL * Multi-AZ * Instance Class: db.m5.large * Storage: SSD (gp3) 500 GiB * Snapshot retention: Daily system snapshot 7 days (Default) * [Pricing](https://aws.amazon.com/rds/pricing/) #### AWS Elastic Container Service (ECS) Fargate * **Cost Factors** * ECS Task vCPU/Memory allocation * Ephemeral Storage > 20GB * **JHU Configuration** * 3 ECS Tasks run for deposit and notification services 24/7 * Per task 1vCPU/4GBMem * 4 ECS tasks run for data loader jobs for 2 hours/day * Per task 2vCPU/4GBMem * 20GB ephemeral storage for each Task * [Pricing](https://aws.amazon.com/fargate/pricing/) #### AWS Elastic Compute Cloud (EC2) * **Cost Factors** * EC2 Instance Type * Storage Size/IO Bandwidth * Data Transfer * **JHU Configuration** * 1 EC2 for Pass-core/Pass-UI, Instance Type: m5.large, Storage: EBS 40GiB * 1 EC2 for jumpbox, Instance Type: t3.micro, Storage: EBS 8GiB * [Pricing](https://aws.amazon.com/ec2/pricing/) #### AWS CloudWatch * **Cost Factors** * Size of Logs * Number of Custom Metrics * Number of Dashboards * **JHU Configuration** * 7 alarms * 3 custom metrics * 1 Dashboard * 1 Canary Script * [Pricing](https://aws.amazon.com/cloudwatch/pricing/) #### AWS Load Balancer * **Cost Factors** * Usage rate * Data transfer * **JHU Configuration** * 2 ALBs * 3 Listeners and Rules * [Pricing](https://aws.amazon.com/elasticloadbalancing/pricing/) #### AWS Web Application Firewall (WAF) * **Cost Factors** * Number/type of Rules * Bot Control * **JHU Configuration** * 3 Rules * 1 Bot Control * [Pricing](https://aws.amazon.com/waf/pricing/) #### AWS Simple Storage Service (S3) * **Cost Factors** * Storage Type * Storage Size * **JHU Configuration** * \~ 5 Buckets * \~ 3 GB Total as of Sept 2024 * [Pricing](https://aws.amazon.com/s3/pricing/) ### AWS Cost Metrics Sampling JHU’s PASS production architecture detailed above results in a fairly-consistent operating cost totaling $700-$800 per month.
PASS Account Spend

PASS Account Spend

* About half of the spend is dedicated to the High Availability (HA) production database. * Compute resources comprise roughly a quarter of the spend. * The remainder of the cost is spread across supporting services, such as monitoring logging, and security. * Storage costs for PASS are currently negligible. ## Sample of Cost Metrics: July, 2024 | Service | Costs | | -------------------------- | ----- | | **RDS** | $379 | | **ECS** | $134 | | **EC2** | $76 | | **CloudWatch** | $50 | | **EC2-Other** | $39 | | **Elastic Load Balancing** | $33 | | **WAF** | $15 | | **VPC** | $14 | | **CloudTrail** | $12 | | **Config** | $3.82 | | **DevOps Guru** | $3.12 | | **S3** | $1.54 | | **Lambda** | $1.04 | | **SQS** | $0.24 | | **SSM** | $0.19 | | **ECR** | $0.18 | | **Route 53** | $0.10 | | **Key Management Service** | $0.06 | | **Total costs** | $772 | # PASS Versioning A release of PASS is made up of Java artifacts and Docker images. Maven builds all the Java artifacts and the associated Docker images. The Node based pass-ui is released as a Docker image. Docker images are published in GitHub Container Registry which can be viewed in each repository's Packages. There is a single version of PASS across all components. We've decided to take this approach due to several benefits, but the most important reason is the ease of understanding. PASS uses semantic versioning following this convention: ``` X.Y.Z-[other-labels] ``` Where, * X = Major updates * Y = Minor updates * Z = Patches (bug fixes) The other labels include: * SNAPSHOT, the current development release. * RC\[X] e.g. RC1 or RC2, which stands for Release Candidate 1, Release Candidate 2 etc. The SNAPSHOT label is generated for the next development version and happens automatically in our CI/CD pipeline. When a release is deployed, a SNAPSHOT is automatically created during the release. The release candidate label is preparing a release to be deployed before the final version is completed. Bug fixes are the only updates permitted in a release candidate version. A couple of examples below demonstrate the usage of this versioning scheme: * Release version: ``` 1.9.0 ``` * Patch: ``` 1.9.1 ``` * Development version: ``` 1.10.0-SNAPSHOT ``` * Release Candidate 1: ``` 1.10.0-RC1 ``` # How to Deploy In this article the JHU instance of PASS will be used to demonstrate how we deploy the application to production. As with the JHU architecture decisions, deployment will be done in various ways and will likely vary between institutions depending on their architecture. It is important to note that there is a process of testing and stepping through development and staging environments before deploying to production. This articles will focus on one deployment workflow to a production environment. ## GitHub (GH) Automations [GitHub Actions](https://docs.github.com/en/actions) is used to deploy PASS to our production environment. We keep the GH Workflow YAML and Python scripts in a private repository since this is a JHU specific deployment. The main workflow `Production Deployment` is run from the GH web interface and releases a specified version of the main branch from the PASS repositories. The only parameter required to run the workflow is the release version. The `Production Deployment` workflow is responsible for initiating the entire deployment process which encapsulates other workflows. The following GH workflows are part of the deployment: 1. `Production Deployment:` * This is the main production deployment workflow, which orchestrates the deployment of various components of an application to the production environment. It is triggered manually with a specified release version. The workflow consists of multiple jobs that utilize two workflows (`Publish to SNS Topic: Triggers Deployment to AWS` and `Deploy PASS Services and Data Loaders to ECS AWS`) to deploy different parts of the application, such as PASS Core, Deposit Services, Notification Services, and the Data Loaders. Each job specifies its environment as `Production` and uses the release version as the Docker image tag. The release version is set by clicking the `Run workflow`button and specifying the `Release Version`. 2. `Publish to SNS Topic: Triggers Deployment to AWS:` * This workflow is designed to deploy an application to AWS by triggering an AWS SNS (Simple Notification Service) topic. It accepts inputs such as environment, commit reference, secrets, and Docker image tag. The workflow includes steps to checkout the repository, set up Python, install necessary Python packages (such as boto3 for AWS interactions), and run a Python script that publishes a message to the SNS topic, initiating the deployment process in AWS. 3. `Deploy PASS Services and Data Loaders to ECS AWS:` * This workflow focuses on deploying specific services and data loaders to AWS ECS (Elastic Container Service). Similar to the previous workflow, it takes inputs like environment, Docker image name, secrets, and Docker image tag. The steps involve checking out the repository, setting up Python, installing necessary packages, and running a Python deployment script that deploys the specified services and data loaders to AWS. ### Steps Performed During the Deployment to Production 1. **Trigger the Deployment**: * The deployment to production is initiated manually through the `workflow_dispatch` event, requiring a specified release version. 2. **Deploy PASS Core/UI Application**: * The `deploy_pass_app` job uses the `Publish to SNS Topic: Triggers Deployment to AWS` workflow to deploy the core application to the production environment. It provides the necessary inputs, including environment (`Production`), commit reference, and Docker image tag, all set to the specified release version. 3. **Deploy Deposit Services**: * The `deploy_deposit_services` job uses the `Deploy PASS Services and Data Loaders to ECS AWS` workflow to deploy the deposit services to AWS ECS. It specifies the Docker image name (`deposit-services-core`) and tag as the release version. 4. **Deploy Notification Services**: * The `deploy_notification_services` job also uses the `Deploy PASS Services and Data Loaders to ECS AWS` workflow to deploy notification services. It sets the Docker image name to `pass-notification-service` and the tag to the release version. 5. **Deploy Grant Loader**: * The `deploy_grant_loader` job deploys the grant loader using the same workflow, with `jhu-grant-loader` as the Docker image name. 6. **Deploy Journal Loader**: * The `deploy_journal_loader` job deploys the journal loader, specifying `pass-journal-loader` as the Docker image name. 7. **Deploy NIHMS Loader**: * Finally, the `deploy_nihms_loader` job deploys the NIHMS loader, using `pass-nihms-loader` as the Docker image name. Each job inherits secrets necessary for AWS access and uses specific Docker images corresponding to different components of the application, all tagged with the release version to ensure consistency across the deployment. The Docker images final destination is the Elastic Container Registry (ECR) in the [PASS application architecture](/infrastructure-documenation/operations-production/ops-aws-arch#pass-elastic-container-registry-ecr). # Monitoring ## Alarms There are several alarms setup in CloudWatch to monitor the health of the PASS system. Triggered and cleared alarms use an SNS topic to send email to PASS devops. The following are a subset of Alarms to monitor the PASS application: * **Relational Database Service (RDS)** * **CPUUtilization** - The CPU Utilization of the RDS instance. Normally will alarm if this metric is higher than some threshold for an amount of time. * **Elastic Compute Cloud (EC2)** * **CPUUtilization** - The percentage of physical CPU time that Amazon EC2 uses to run the EC2 instance, which includes time spent to run both the user code and the Amazon EC2 code. * **CloudWatch Log Errors** - Monitor will alarm if an ERROR log entry appears in PASS logs in cloudwatch. * **Elastic Cloud Service (ECS)** * **MemoryUtilized** - The memory being used by tasks in the resource that is specified by the dimension set that you're using. * **CpuUtilized** - The CPU units used by tasks in the resource that is specified by the dimension set that you're using. * **NetworkTxBytes** - The number of bytes transmitted by the resource that is specified by the dimensions that you're using. This metric is obtained from the Docker runtime. * **CloudWatch Log Errors** - Monitor will alarm if an ERROR log entry appears in PASS logs in cloudwatch. * **Application Load Balancer (ALB)** * **HTTPCode\_ELB\_4XX\_Count** - The number of HTTP 4XX client error codes that originate from the load balancer. * **TargetResponseTime** - The time elapsed, in seconds, after the request leaves the load balancer until the target starts to send the response headers. * **UnHealthyHostCount** - The number of unhealthy instances registered with your load balancer. An instance is considered unhealthy after it exceeds the unhealthy threshold configured for health checks. * **Simple Queue Service (SQS)** * **NumberOfMessagesSent** - The number of messages added to a queue. * **ApproximateAgeOfOldestMessage** - The approximate age of the oldest non-deleted message in the queue. * **Simple Notification Service (SNS)** * **NumberOfNotificationsFailed** - The number of messages that Amazon SNS failed to deliver. There are a lot more metrics that can be collected and fine-tuned with different parameters. These are a sample of metrics that JHU PASS uses, but it is encouraged to see what metrics and their parameters that best fit the environment PASS is in. More information regarding CloudWatch and the metrics that can be monitored are on the [AWS documentation site](https://docs.aws.amazon.com/). ## Synthetic Canary Amazon CloudWatch Synthetics are configurable scripts than run a schedule and can monitor endpoints and APIs. Canaries follow the same route and patterns as someone using the application. This enables the ability to detect problems before they happen. The JHU PASS configuration has a simple CloudWatch Synthetic Canary script that is calling the application every 10 minutes. If a 403 for the login page is not returned, an alert will be triggered sending an email to PASS devops. Learn more about setting up a synthetic canary on the [Cloud Watch User Guide](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Synthetics_Canaries.html). ## Logs CloudWatch Logs are used for logging from all PASS Docker containers. The Access Logs have been enabled on the Public ALB. These logs are written to an S3 bucket. In order to view/query the access logs, an AWS Athena database has been created. You can query the Athena database using standard SQL in the Query Editor. ## Dashboard A dashboard has been added to CloudWatch that shows high level metrics for the PASS app, triggered alarms, and errors in logs. To learn more about CloudWatch dashboards visit the [CloudWatch User Guide](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Dashboards.html.) ## Data Loader Failure Notifications It is important to know when one of the PASS Data Loader jobs fail to run. This can be done in EventBridge by creating a Rule that fires on AWS Batch Job failures and sends a notification using AWS SNS. # Data Loaders This section describes how the [Data Loaders](/developer-documentation/data-loaders) are implemented in the JHU AWS production environment. ## Journal Data Loader * **AWS Batch:** The Journal Loader is executed as a batch job using AWS Batch. The Journal Loader batch job retrieves, processes, and uploads journal data to the PASS system. * **ECS:** The batch job is executed in ECS Fargate within AWS. The environment is deployed in the main VPC. * **EventBridge:** AWS EventBridge is used to schedule the execution of the Journal Loader job. ## Grant Data Loader * **AWS Batch:** The Grant Loader is executed as a batch job using AWS Batch. The Grant Loader batch job pulls grant data from the JHU grant management databases that is used by Johns Hopkins University. * **ECS:** The batch job is executed in ECS Fargate within AWS. The environment is deployed in the main VPC and JHU connected VPC. * **S3:** The configuration of the Grant Loader is persisted in S3 buckets. The policy information `policy.properties` and `grant_update_timestamps` are stored in a S3 bucket. * **EventBridge:** AWS EventBridge is used to schedule the execution of the Grant Loader job. The target of the schedule is a Step Function which breaks out the pulling and loading of data into separate steps. It also defines functionality for retrying failed loads with a certain number of attempts and interval wait time. ## NIHMS Data Loader * **AWS Batch:** The NIHMS Loader is executed as a batch job using AWS Batch. It pulls data from the NLM’s Public Access Compliance Monitor (PACM) API updating and creating submissions. * **ECS:** The batch job is executed in ECS Fargate within AWS. The environment is deployed in the main VPC. * **EventBridge:** AWS EventBridge is used to schedule the execution of the NIHMS Loader job. ### PACM API Token The NIHMS Data Loader Harvester process requires an NIHMS API Authentication token. This token is available from the NIHMS/PACM utils page and is valid for three months. In order to obtain a NIHMS API Authentication token, you must create an account with NIH/ERA. See this page for instructions on creating a User Account with NIH/eRACommons: [Create Commons Account](https://www.era.nih.gov/erahelp/ams_new/Content/Create_Accounts/Create_User_Accts/Create_Acct_External.htm) * System Owner: The system owner is the NIH, but account management is delegated to a University’s Office of Sponsored Research. In JHU’s case, this is: [Johns Hopkins University Research Administration](https://jhura.jhu.edu/). For any other university setting up their own NIHMS data loader, it will be the Office of Sponsored Research that creates the account for the API key. * Account Setup: In the case of JHU, the account needs to be set up by [JHU Research Administration](https://jhura.jhu.edu/). They will need to have the following permissions in order to create an account: SO, AO, AA, or BO and they cannot be affiliated with more than one institution. If any modifications to the account need to be made, such as changing the associated email with the account, it will need to be done via the eRA administrator. More information regarding the NIHMS Loader can be found in the [Developer Documentation NIHMS Loader section](/developer-documentation/data-loaders/nihms-loader) ### NIHMS API Token Refresh Automation There currently is no NIHMS/PACM API available to refresh the NIHMS API authentication token. Obtaining/refreshing the token must be done using the NIHMS PACM website. PASS has created a Robotic Process Automation (RPA) to generate a new API Token and set the token into a store. The default store is an AWS Parameter Store. The Token is generated by running a script that logs into the PACM utils page using th ERA Commons login, clicks on the API Token link, and writes the new token to a file named in the `NIHMS_OUTFILE` environment variable. The value in the `NIHMS_OUTFILE` file is then set into the parameter store. There is a Docker image available named `ghcr.io/eclipse-pass/pass-nihms-token-refresh:`. To run the token refresh RPA, the following needs to be passed to the docker image as environment variables: * `NIHMS_USER` : The ERA Commons login username * `NIHMS_PASSWORD` : The ERA Commons login password * `NIHMS_OUTFILE` : The full path to the file to write the new token ``` docker run -e NIHMS_USER= -e NIHMS_PASSWORD= -e NIHMS_OUTFILE= ghcr.io/eclipse-pass/pass-nihms-token-refresh: ``` If you are setting the token into AWS Parameter Store, the RPA assumes that the needed AWS CLI authentication are in place such as passing AWS keys as environment variables or using IAM roles. # Data & Backups ## Data that is stored by PASS The data in the PASS application is the submissions, deposits, grants, journals, users, policies, and various PASS data model objects. In addition, there are configuration files that detail how different services and data loaders are run. ## Where is the Data Stored? The data for the PASS data model is stored in Amazon Relational Database Service (RDS), with the exact implementation being a PostgreSQL database. The configuration files are stored in S3 buckets as well as temporary file storage for files uploaded through the Pass Core File Service. These files managed by the File Service are stored in an [Oxford Common File Layout (OCFL)](https://ocfl.io/) format that allows for an independent way of storing files in a structured, transparent and predictable fashion. ## How the Data is Protected and Backed-up AWS RDS supports automated backups. These backups occur daily and include snapshots of the entire database instance, enabling point-in-time recovery to any second within the retention period. Moreover, manual snapshots of the RDS instance can be created at any time. These snapshots are user-initiated and are retained until explicitly deleted. S3 buckets are used to store configuration files and metadata in the PASS application. AWS S3 buckets supports versioning. When versioning is enabled on an S3 bucket, AWS retains multiple versions of an object. This feature allows recovery from accidental overwrites or deletions by restoring previous versions. By using AWS S3 buckets it provides [data durability](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html) by ensuring that stored data is highly protected against loss or corruption. It achieves this by replicating data across multiple devices in at least three distinct Availability Zones within an AWS Region. This redundancy helps S3 preserve data even in the case of hardware failures or the loss of an entire Availability Zone. There are also routine procedures to verify the integrity of the data using checksums. Additionally, features that can be managed such as versioning, object lock, and cross-region replication further enhance data protection by safeguarding against accidental or malicious deletion and enabling disaster recovery. # Eclipse Operations PASS is an Eclipse Foundation project and benefits from the resources and knowledge that the Eclipse Foundation has to offer. One of those resources is the Eclipse GH repository configuration. ## Eclipse GitHub Repository Configuration The [.eclipsefdn](https://github.com/eclipse-pass/.eclipsefdn) repository enables the team committers to [self-service several aspect of the eclipse organization](https://www.eclipse.org/projects/handbook/#resources-github-self-service) via a tool called [Otterdog](https://otterdog.readthedocs.io). These [.eclipsefdn](https://github.com/eclipse-pass/.eclipsefdn) repo / tools gives access to: * Organization settings * Organization webhooks * Repositories and their settings * Branch protection rules If a setting is not supplied in the `eclipse-pass.jsonnet` file, it will default to a predefined value. The full set of default values is available in the Eclipse-managed [otterdog-defaults.libsonnet](https://github.com/EclipseFdn/otterdog-defaults/blob/main/otterdog-defaults.libsonnet). Learn more about [Otterdog here](https://otterdog.readthedocs.io/en/latest/). ### Workflow for Updating the Otterdog Configuration To make changes to the Otterdog configuration in the [.eclipsefdn](https://github.com/eclipse-pass/.eclipsefdn) repository, follow these steps: 1. Fork the [.eclipsefdn](https://github.com/eclipse-pass/.eclipsefdn) repository into your own GitHub account. 2. Make changes to the `eclipse-pass.jsonnet` file. 3. Push those changes to the upstream repository. 4. Create a pull request in the `.eclipsefdn` repository. 5. An automated workflow will run, displaying the changes to be applied and validating that the configuration is correctly formatted and structured. 6. Depending on the type of changes one or more Eclipse engineers will review the PR. Additionally, the project lead's approval may also be required based on the nature of the changes. ## Eclipse Contributor Agreement and Eclipse Development Process Contributors of the project must electronically sign appropriate documents in order to become committers. The following are agreements and policies by Eclipse that a committer must read: * [Eclipse Contributor Agreement (ECA)](https://www.eclipse.org/legal/eca/) * [Eclipse Development Process (EDP)](https://www.eclipse.org/projects/dev_process/) ## ECA for Pass Documentation ECA is not configured as a required check for merging in `pass-documentation`, therefore PRs can be merged with a non-committer. In addition, the EDP explicitly states: "you can merge if you know that the user associated with the commit has signed an ECA", therefore if a user with a different GitHub account with a different email address from their Eclipse committer account, is still able to merge PRs with those commits. This applies to GitBot (GitBook GitHub bot), and Eclipse recognizes this account for making commits. # Next Steps / Institution Configuration Implementing and configuring PASS involves deploying the core application infrastructure using a variety of services provided by AWS. The institution must integrate with authentication systems, configure automated data loaders, set up backups, and monitor the application for performance and security. This guide provides a general outline of what must be done to replicate a similar JHU infrastructure. * **Infrastructure Setup** * **VPC Setup:** * Create a VPC with both public and private subnets for deploying PASS. The public subnet should be used by the public ALB, while private subnets should host the PASS Docker images and other services like the database. * Ensure that security groups are defined to manage inbound and outbound traffic securely. * **EC2 and ECS for Core Services:** * Use EC2 instances to deploy PASS Core, PASS UI, and other related components. * **RDS for Database:** * Deploy the PostgreSQL database for PASS using Amazon RDS. This database will store submission data, policies, grants, user information, and other metadata. * **S3 for Storage and Configuration Files:** * S3 is used for storing configuration files and temporary file storage submission artifacts. * **Application Load Balancers:** * Configure ALBs to handle incoming traffic for the PASS Core/UI and ensure that traffic is routed to the correct backend services. * There are two target groups, one for the public ALB and one for the private ALB, these will forward the appropriate container that is run in the PASS Core EC2 instance. * **SQS/SNS:** * SQS queues are used throughout the PASS application architecture. Notably they are used for messages related to submissions, deposits, and submission events. * Configure notifications via SNS for alerts triggered by CloudWatch. * **Security and Authentication** * **IAM Roles and Policies:** * Define and assign IAM roles to control which services can access different resources. For example, assign roles for the ECS tasks to interact with S3, RDS, SQS, and other AWS services. When configuring the IAM role policies, the recommendation is to follow best practices of [least privilege](https://aws.amazon.com/blogs/security/techniques-for-writing-least-privilege-iam-policies/). * **SSL and TLS** * Use AWS Certificate Manager (ACM) to manage SSL/TLS certificates for secure communication between users and the PASS application. These certificates are attached to the load balancers. * **SSO Integration** * JHU is integrated with Shibboleth SSO, which enables JHU users to login with their institutional credentials. It is possible to integrate with other SSO solutions for authentication, but may require some additional development to handle the header variables. * Web Application Firewall (WAF) * Configure a WAF using a set of web ACLs that adhere to your institution's web security practices. AWS provides a good set of predefined rules that can be applied. * **Data Loaders** * **Batch Jobs** * AWS Batch is composed of a Compute Environment, Job Queue, and Job Definition. AWS Batch uses the Job Definition to run the PASS Data Loader job in the required VPC associated with the compute environment. * **Scheduling** * **EventBridge** * Schedulers are used to run the Grant, Journal, and NIHMS Batch Jobs. The schedule will be dependent on the institution, but the JHU implementation the Grant and NIHMS jobs run daily and the Journal job on a weekly basis. * **Backup and Disaster Recovery** * **RDS Backups** * RDS has the capability to automate backups and provides a point-in-time recovery capabilities. * **S3 Versioning:** * Enabling version on S3 buckets will allow for data recovery in case of accidental deletions or overwrites. * **Monitoring and Logging** * **CloudWatch** * Set up alarms and metrics to track CPU usage, memory utilization, and network traffic for various AWS resources. Some example alarms have been mentioned in the [Monitoring section](/infrastructure-documenation/operations-production/ops-monitor#alarms). The exact alarms to be created are for the implementing institution. * **SNS** * By enabling notifications via SNS, alerts from CloudWatch can be sent to phone or email. # Welcome to the Public Access Submission System (PASS) Documentation Welcome to the Public Access Submission System (PASS) Documentation! If you're a first-time visitor, we recommend beginning with our Welcome Guide, accessible that you start with the welcome guide [here](/pass-documentation-dev/welcome-guide). This guide offers a comprehensive introduction to PASS, detailing the unique challenges it addresses for researchers, guiding you through the initial setup, and providing a detailed overview of the PASS architecture. Dive in to discover how PASS simplifies and streamlines the submission process for your research! If you're already familiar with PASS and want jump right in, you can find our developer docs [here](/pass-documentation-dev/developer-documentation) or our infrastructure docs [here](/pass-documentation-dev/infrastructure-documenation) # PASS Welcome Guide Welcome to the Public Access Submission System (PASS) documentation! The Public Access Submission System (PASS) is an Eclipse open source platform designed to assist researchers, IT staff, compliance officers, and executives in efficiently and economically complying with the access policies of their funders and institutions. Use Eclipse PASS to submit your manuscripts to funder and institutional publication repositories (e.g. PubMed Central, DSpace) and comply with access policies. Using the web-based PASS system, you can avoid paying article processing charges to make your publication open to the public, simultaneously send your manuscript to multiple repositories seamlessly, populate forms automatically with publication and author information by providing DOIs and ORCID IDs. In this guide we step through various topics on PASS: * Research submissions overview * Problems Researchers face when submitting to different repositories * How PASS is solving submission challenges for researchers * PASS at JHU * PASS demonstrations at Conferences * Technology stack * Deployment architecture * Latest release * Setup and run PASS * Collaboration with other institutions * Contributing to PASS # Research Submission Overview The journey of research from conception to dissemination is a long one. After designing your study, gathering and analyzing your data, undergoing peer review, there is still more to be done. For many researchers the next stage is submitting your manuscripts and all the supporting documents to a variety of repositories, as dictated by the requirements of funding bodies and academic institutions. ## Challenges Faced by Researchers Identifying the appropriate repositories for submission is not always straightforward. Researchers often juggle multiple grants, each with its unique set of requirements. Gathering all this information can be time consuming and using many systems can be tedious and frustrating. The workflows for each system can be complex and navigating these systems can be burdensome. Additionally, there can be processing charges for each repository, which are often incurred on a per-repository basis, cannot be overlooked. ## PASS: Solving submission challenges The Eclipse Public Access Submission System (PASS) is a web-based system that aims to solves these challenges. PASS is engineered to support researchers, IT personnel, compliance officers, and executives in meeting the access policies of their funders and institutions both efficiently and economically. By leveraging Eclipse PASS to submit your manuscripts to both funder and institutional publication repositories (such as PubMed Central, DSpace) your workflow becomes streamlined, ensuring compliance and avoiding costly article processing charges to make your publication open to the public. By simply signing into PASS, it gathers all the relevant information about your grants, publications, and previous submissions to populate forms with publication and author information by providing DOIs and ORCID IDs. It simultaneously sends your manuscript to multiple repositories seamlessly with a few single clicks! Our mission is to support researchers, and we know the frustrations and challenges researchers face trying to get their research easily accessible in the public domain. PASS was originally built in collaboration by the Eclipse Foundation and the Digital Research Curation Center at Johns Hopkins Sheridan Libraries. As an evolving open-source project, PASS welcomes the collaboration of universities and institutions, aiming to continually refine and enhance a platform that frees up researchers to do what they're passionate about: pushing the boundaries of what we know. # PASS at JHU Since its inception in 2018, PASS at Johns Hopkins University (JHU) has been a vehicle for research submission and compliance. In 2023, we proudly introduced our 1.0 release, featuring a major overhaul of our back-end architecture. This update introduced a robust and well-defined API, streamlined our release process, aligned dependencies, and restructured components for easier system management. Designed to fit the needs of the Office of Science and Technology [memo ](https://www.whitehouse.gov/wp-content/uploads/2022/08/08-2022-OSTP-Public-Access-Memo.pdf)and aligns with JHU’s own Open Access Policy, PASS ensures that all JHU researchers can effortlessly publish their work to JHU's institutional repository, [JScholarship](https://provost.jhu.edu/faqs/what-is-jscholarship/). Beyond meeting JHU’s requirements, PASS at JHU facilitates submissions to PubMed Central through the [NIH Manuscript Submission](https://www.nihms.nih.gov/) (NIHMS) system. Looking ahead, we’re excited to expand our support to additional repositories, such as NSF. Take a look at our [PASS Roadmap](/pass-documentation-dev/community/pass-roadmap) article for a glimpse into the future features and enhancements in the pipeline. Architecturally, PASS is built on Amazon Web Services and leverages a Java Spring Boot web application paired with a REST API for seamless core service integration. While we’ve chosen AWS for its reliability and scalability, it’s essential to note that PASS’s flexibility allows it to thrive on various platforms. For an in-depth look at our system’s blueprint, we invite you to explore our [Deployment Architecture](/pass-documentation-dev/welcome-guide/deployment-architecture) article. # PASS Demonstrations at Conferences ### Eclipse Pass Project Briefing, January 2024 In 2023, the PASS team at Johns Hopkins University achieved significant milestones including a series of monthly releases that introduced a new back-end architecture, enhanced API, and a range of system improvements to support a more efficient and user-friendly experience. Alongside development, the team engaged in community building with academic partners and hosting providers to explore piloting opportunities and integration needs for 2024. The PASS project plans to focus on expanding user engagement, building a broader community of institutions interested in utilizing PASS, enhancing system administration tools, and making application improvements, including better integration with funder repositories. For more details visit the [Project Briefing](https://drive.google.com/file/d/1WvAUQLLXbGAsCjBDgjUK-so0glmjwsaF/view) *** ### CNI Presentation, December 2023 {% embed url="" %} PASS CNI Presentation December 2023 {% endembed %} For more details about this presentation visit the [CNI website](https://www.cni.org/topics/ci/the-nsf-public-access-initiative-projects-funded-and-catalytic-aims-of-the-program) *** ### Eclipse PASS Project Briefing, January 2023 In 2022, the Public Access Submission System (PASS) project embarked on significant advancements, notably transitioning to an Eclipse Foundation open-source project, and upgrading its server architecture. This strategic move to the Eclipse Foundation enhanced PASS with engineering, community management, and organizational governance expertise, enabling more effective collaboration with agencies and institutions. Furthermore, the architectural upgrade enhanced the system's API, simplified deployment, and streamlined system management, setting the stage for broader community engagement and institutional collaboration. For more details visit the [Project Briefing](https://drive.google.com/file/d/12jCpURDYDbfiAnzjBMeukL1zSBsf-hB8/view) *** ### CNI Presentation, December 2022 {% embed url="" %} PASS CNI Presentation December 2022 {% endembed %} View the [Presentation Slides](https://www.cni.org/wp-content/uploads/2022/12/Eclipse-PASS-CNI-Presentation-2022-12-13-Bill-Branan.pdf), or more details about this presentation visit the [CNI Website](https://www.cni.org/topics/ci/how-the-public-access-submission-system-is-ideally-suited-to-address-the-new-ostp-memorandum) *** ### CNI Presentation, Spring 2020 {% embed url="" %} PASS CNI Presentation Spring 2020 {% endembed %} For more details about this presentation visit the [CNI Website](https://www.cni.org/topics/repositories/packaging-specification-for-simultaneous-deposit-of-articles-and-data-into-multiple-repositories) *** ### FORCE11 Presentation, November 2018 {% embed url="" %} PASS FORCE11 Presentation November 2018 {% endembed %} View the [Presentation Slides](https://zenodo.org/records/1453344), or for more details about the presentation visit the [FORCE11 Website](https://force11.org/force2018/) # Technology Stack ### Introduction PASS is composed of four main components: PASS UI, Data Loaders, PASS Core, and Deposit Services. These components are integral to the application running, but are flexible enough that they can be stood up on their own. Built with an array of open-source technologies, PASS guarantees a sturdy, secure, and transparent framework for deployment across diverse environments. PASS UI contains the Submission UI which is responsible for the researcher workflow to review their grants and create submissions to their designated repositories, all while adhering to the policies established by the PASS Core's policy engine. PASS UI interfaces with PASS Core through Shibboleth authentication to support a single sign-on (SSO) experience, facilitating secure and streamlined access to the system's core functionalities. PASS Core is the central back-bone to the application which is responsible for orchestrating the entire researcher workflow by communicating with external services, managing the data layer, applying policy and metadata rules. Data pertaining to journals, grants, and publications is funneled into PASS Core via Data Loaders, where it is then managed by the Java Persistence API (JPA). PASS Core also employs the Oxford Common File Layout (OCFL) for the storage of submission-related documents, offering options for disk or Amazon S3 bucket retention. When researchers are performing their submissions, it will flow through the PASS API. From there, a submission message is placed in a publication queue which the deposit services will pick up, assemble the deposit and transport it to their respective repository (DSpace, PubMed, etc).

PASS Architecture

### Front-end Technologies To support the Submission UI the following technologies and frameworks are employed by PASS: * [Ember.js](https://emberjs.com/): Selected for its robustness and opinionated framework structure, the Submission UI uses Ember.js to provide a clear, consistent, and easy to use workflow. The UI is written in TypeScript using Glimmer Template Syntax (GTS) components. * [Vite](https://vite.dev/): The build tool and development server for pass-ui, providing fast hot module replacement (HMR) during development and optimized production builds. * [WarpDrive](https://warp-drive.io/): The data layer (formerly Ember Data) that manages communication with the JSON:API backend via a request handler chain. ### Back-end Technologies The back-end is written in Java and uses the following technologies and frameworks: #### Pass Core * [Spring Boot](https://spring.io/projects/spring-boot): This is the back-bone of PASS, and the framework driving the development. Spring Boot simplifies the bootstrapping of PASS, by providing a methodology of convention over configuration for cleaner code and easier deployment. * [Elide](https://elide.io/): Exposes a JSON-API web service, which enables a versatile and standardized API for communication to the core logic of PASS. * [JPA](https://spring.io/projects/spring-data-jpa): The data access layer that communicates with the PostgresSQL database. Having Spring automatically wire up the interface to the database reduces boilerplate code and produces clean data access code. * [OCFL](https://github.com/OCFL/ocfl-java): An open implementation in Java, OCFL is used within PASS' file service to reliably store documents. It has support for both filesystem storage and AWS S3 integration. * [PostgreSQL](https://www.postgresql.org/): Serves as the database that stores all the information associated with grants, journals, publications, submissions, and policies. #### Data Loaders & Deposit Services In addition to using Spring Boot the Data Loaders and Deposit Services use the following technologies: * [SWORD](https://sword.cottagelabs.com/): This protocol is specifically utilized for the automated deposit of digital content into repositories. The SWORD protocol within PASS enables the standardized submission of scholarly works. In effect, providing compatibility and ease of integration with a variety of other repository platforms. * [Amazon SQS](https://aws.amazon.com/sqs/): Queues the deposit requests, allowing for asynchronous processing and ensuring that submissions are handled reliably. This decouples the submission process from the deposit execution, enhancing system resilience and scalability. ### Deployment and Hosting PASS is designed to be flexible and run on a variety of platforms; however at JHU we host PASS on Amazon Web Services. You can find more details about our deployment on our [Deployment Architecture](/pass-documentation-dev/welcome-guide/deployment-architecture) page. ### Development Tools and Practices * [Vite](https://vite.dev/) / [Ember CLI](https://cli.emberjs.com/release/): Vite is used as the build tool and dev server for pass-ui. Ember CLI provides scaffolding and test runner support. * [Docker](https://www.docker.com/): Utilized for running and testing PASS locally. * [TestCafe](https://testcafe.io/): The main framework of our PASS acceptance tests. It is a free and open-source solution for running end-to-end tests. * [Test Driven Development](https://en.wikipedia.org/wiki/Test-driven_development) & [JUnit](https://junit.org/): The Eclipse PASS team follows test driven development (TDD) to ensure high code quality and build confidence. Additionally, we use JUnit for unit tests and [Spring Boot Test](https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#features.testing)/[Testcontainers](https://testcontainers.com/) for integration tests. * [GitHub](https://github.com/): GitHub is where the source code for PASS is hosted. GitHub is also used to trigger deployments using a feature called GitHub Actions. # PASS Architecture PASS is designed to run on a variety of different platforms and architectures capable of running Spring Boot applications. This article describes one deployment strategy by highlighting the own Johns Hopkins University's application and deployment architecture. To achieve this deployment, several techniques are utilized including Continuous Integration and Continuous Deployment as known as CI/CD. ## PASS Application Architecture The PASS system's Amazon Web Services (AWS) application architecture employs a variety of AWS services to ensure high availability, security, and scalability. Two main components of the application architecture are the Elastic Cloud Compute (EC2) instances, which host the main application and the Elastic Container Service (ECS) containers, which are composed of the supporting Deposit services, Notification Services and Data Loaders. The diagram below provides a visual representation of JHU's deployment architecture:

JHU's Application Architecture

The main PASS application utilizes Elastic Compute Cloud (EC2) instances for Pass-core API and the Pass-UI. An Application Load Balancer (ALB) sits between these instances and Web Application Firewall (WAF), which gives security and stability to the PASS application. The ALB distributes incoming requests to appropriate target groups, which in turn route traffic to the correct EC2 instances running Pass-core and Pass-UI. The configuration of the Docker images running in EC2 comes from an S3 bucket which stores the configuration files. Persistence of the PASS application data is achieved by AWS Relational Database Service, and in particular using a PostgreSQL instance. All Docker images for the EC2 instances are stored in the Elastic Container Registry (ECR). The PASS application submits messages to the Simple Queue Service (SQS) when a submission is made. The Deposit Services listens for this and consumes those messages. This configuration, with the EC2 instances support a scalable and flexible environment when new application features are implemented and when user demand grows. Supporting the application infrastructure, AWS ECS Fargate hosts containerized services, namely the Deposit and Notification Services, which handle background processing and user interaction through scheduled tasks and real-time event responses. The AWS Batch service running on ECS, with dedicated computational environments for batch jobs, allows for efficient management of tasks such as data transfer from institutional repositories and JHU data sources. As mentioned earlier, the Deposit Services listens to the SQS and consumes the messages in the event of a submission and makes the appropriate deposit to the relevant institutional repository. This comprehensive setup ensures that the PASS system remains responsive and capable of scaling according to demand, while also adhering to organizational policies and data governance standards. ## PASS Deployment Architecture The deployment workflow starts when developers contribute code to the Eclipse PASS Git repository. Changes in this repository trigger GitHub Actions workflows, which are part of an automated CI/CD pipeline facilitating the deployment of the updated code. In deployment SQS is utilized for initiating the deployment from a GitHub workflow that publishes a SNS topic to the queue. The PASS deployment files contain configurations and environment variables that assist in the deployment of the PASS application and supporting services. Liquibase is utilized to manage database schema changes, which interacts with an AWS RDS instance running PostgreSQL.

JHU's Deployment Architecture

For other organizations looking to adopt a similar AWS application and deployment model, it's important to recognize that while the core architecture offers a template, it should be adapted to meet an organization's own requirements and needs. Each organization will need to evaluate its own application demands, data sensitivity, and user base to tailor the cloud resources, network configurations, and security policies accordingly. Moreover, integrating other types of cloud infrastructure or even on-premise solutions might be necessary to address specific technological preferences or regulatory requirements. Hybrid cloud environments or multi-cloud strategies could be employed to leverage the strengths of various cloud providers, enhance resilience, and avoid vendor lock-in. PASS is designed to be flexible and can adapt to a variety of architectures; whether a cloud infrastructure, hybrid or on-premise. # Latest Release We publish the latest release of PASS Core, Pass Support (Data Loaders, Deposit Services, Notification Services), PASS UI, and other supporting repositories to [GitHub](https://github.com/eclipse-pass). The artifacts from the Java based components are also published to a [Sonatype Maven Central Repository](https://central.sonatype.com/artifact/org.eclipse.pass/eclipse-pass-parent). Release announcements are made using a [Google Groups](https://groups.google.com/g/pass-general) mailing list, these announcements provide a brief summary highlighting the features, updates, and bug fixes that went into the release. ### GitHub * [PASS Parent (aka main)](https://github.com/eclipse-pass/main/releases) * [PASS Core](https://github.com/eclipse-pass/pass-core/releases) * [PASS Support](https://github.com/eclipse-pass/pass-support/releases) * [PASS UI](https://github.com/eclipse-pass/pass-ui/releases) * [PASS Docker](https://github.com/eclipse-pass/pass-docker/releases) * [Pass Acceptance Testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases) * [Pass Documentation](https://github.com/eclipse-pass/pass-documentation/releases) ### Sonatype During our release the Java based component artifacts for PASS are published to Sonatype Maven Central Repository: * [PASS Parent](https://central.sonatype.com/artifact/org.eclipse.pass/eclipse-pass-parent) * [PASS Core](https://central.sonatype.com/artifact/org.eclipse.pass/pass-core) * [PASS Support](https://central.sonatype.com/artifact/org.eclipse.pass/pass-support) ### SBOM As of release `1.13.0`, a [CycloneDX Software Bill Of Materials (SBOM)](https://cyclonedx.org/specification/overview/) is created and published for the following artifacts: * [pass-core-main](https://repo1.maven.org/maven2/org/eclipse/pass/pass-core-main/) * [deposit-core](https://repo1.maven.org/maven2/org/eclipse/pass/deposit/deposit-core/) * [pass-notification-service](https://repo1.maven.org/maven2/org/eclipse/pass/pass-notification-service/) * [pass-grant-loader](https://repo1.maven.org/maven2/org/eclipse/pass/pass-grant-loader/) * [pass-journal-loader-nih](https://repo1.maven.org/maven2/org/eclipse/pass/pass-journal-loader-nih/) * [nihms-data-harvest](https://repo1.maven.org/maven2/org/eclipse/pass/nihms-data-harvest/) * [nihms-data-transform-load](https://repo1.maven.org/maven2/org/eclipse/pass/nihms-data-transform-load/) * [pass-data-client](https://repo1.maven.org/maven2/org/eclipse/pass/pass-data-client/) * [pass-ui (Located in the / directory of the docker image)](https://github.com/eclipse-pass/pass-ui/pkgs/container/pass-ui) # Setup and Run PASS Locally ## Introduction The purpose of this article is to guide you through the process of getting PASS up and running locally using [Docker](https://www.docker.com/). If you're new to Docker, take a look at the [Get Started with Docker Guide](https://www.docker.com/get-started/). There are many other ways to deploy and run PASS, but this article focuses on getting PASS up and running on a local machine for a preview of the application. If looking to deploy PASS into a production environment, please take a look at the [PASS Infrastructure](/pass-documentation-dev/infrastructure-documenation) and [Developer Documentation.](/pass-documentation-dev/developer-documentation) ### Requirements * Docker * Docker Engine version 20.10.21 or higher * A minimum of 16GB of system memory is required, as Docker uses 8.5GB of virtual memory when starting up all services. ## Run PASS Locally ### Simple Setup Running PASS locally requires a few simple steps and no configuration, as defaults and demo data are already provided. If you do want to explore the configuration of PASS, the [configuration section](#configuration) of this page details how to edit the environment file. The first step is to clone or [download ](https://github.com/eclipse-pass/pass-docker/archive/refs/heads/main.zip)the code from the PASS Docker repository: {% code title="Cloning PASS Docker" %} ``` git clone https://github.com/eclipse-pass/pass-docker.git ``` {% endcode %} Once you've either cloned PASS docker or extracted it from the ZIP file, then you will want to use a command line tool to navigate to the root directory of PASS docker. 
cd C:\Users\username\IdeaProjects\pass-docker
Now that you're at the root directory of PASS Docker run the following command: {% code title="Run Docker Compose" overflow="wrap" %} ``` docker compose -f docker-compose.yml -f eclipse-pass.local.yml up -d --no-build --quiet-pull --pull always ``` {% endcode %} After running `docker compose` you should see the following images being pulled and started running in a container. {% code title="Docker Compose Output" %} ``` [+] Running 12/12 - Network pass-docker_front Created - Network pass-docker_back Created - Volume "pass-docker_db" Created - Container ldap Started - Container pass-ui Started - Container auth Started - Container proxy Started - Container localstack Started - Container pass-docker-postgres-1 Started - Container idp Started - Container pass-core Healthy - Container loader Started ``` {% endcode %} After the container is running, PASS is now running locally on your machine! All you need to do now is navigate to using a web browser, such as Firefox, Google Chrome, or Safari. From there you will see a login screen and can login using these test accounts:
User namePasswordDescription
nih-usermooUser that has NIH grants, and submissions on their behalf waiting for approval.
staff1mooUser that has one NIH Grant, and no submissions.
staff2mooUser that doesn't have any grants or submissions.
If you need to restart docker and re-run the containers, use the following command: {% code title="Docker Compose Down" %} ``` docker compose -p pass-docker down -v ``` {% endcode %} This will shutdown docker and remove all data associated with the application. This is important because when running `docker compose up` it loads fake data into the database. If you recreate the containers without previously destroying the data, duplicates will be inserted into the database. ### Advanced Setup The simple setup contains the workflow of creating a submission and simulating the deposit, but it doesn't actually go anywhere. If you want to test the full integration workflow with deposit services and submitting to DSpace or to NIHMS you can run the following command: {% code title="PASS with Deposit Services" overflow="wrap" %} ``` docker compose -p pass-docker -f docker-compose.yml -f eclipse-pass.local.yml -f docker-compose-deposit.yml -f docker-compose-dspace.yml up -d --no-build --quiet-pull --pull always ``` {% endcode %} Add an administrator and sample data into DSpace: {% code title="Add Administrator" overflow="wrap" %} ``` docker compose -p pass-docker -f dspace-cli.yml run --rm dspace-cli create-administrator -e test@test.edu -f admin -l user -p admin -c en ``` {% endcode %} {% code title="Add Sample Data" overflow="wrap" %} ``` docker compose -p pass-docker -f dspace-cli.yml -f dspace-cli.ingest.yml run --rm dspace-cli ``` {% endcode %} With Deposit Services and DSpace running in the container, when you run through the submission process and submit a manuscript to JScholarship it will make a deposit into DSpace. To view a deposit in the locally running DSpace instance navigate to in your web browser. Login using username:`test@test.edu` and password: `admin`. If submitting a deposit to [NIHMS](https://www.nihms.nih.gov), you can view the simulated deposit locally in the `pmc-sftp-server` image that is part of `pass-docker`. The first step is getting the name of the `pmc-sftp-server` container by running the `docker ps` command: ``` docker ps ``` It will output the following: {% code title="Docker ps Output" %} ``` CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c32ca1ae2905 dspace/dspace-solr:dspace-7.6 "/bin/bash -c 'init-…" 41 minutes ago Up 40 minutes 0.0.0.0:8983->8983/tcp dspacesolr fb2d2c0c46c7 ghcr.io/eclipse-pass/deposit-services-core:1.6.0-SNAPSHOT "./entrypoint.sh" 41 minutes ago Up 40 minutes pass-deposit-services 8394fa5b55da ghcr.io/eclipse-pass/idp:1.6.0-SNAPSHOT "./entrypoint.sh" 41 minutes ago Up 41 minutes 4443/tcp, 8443/tcp idp 65e71a8770d4 dspace/dspace:dspace-7.6-test "/bin/bash -c 'while…" 41 minutes ago Up 40 minutes 8000/tcp, 8009/tcp, 0.0.0.0:8080->8080/tcp dspace df73b4336089 ghcr.io/eclipse-pass/pass-core-main:1.6.0-SNAPSHOT "./entrypoint.sh" 41 minutes ago Up 40 minutes (healthy) pass-core 015446cc79c1 ghcr.io/eclipse-pass/pass-auth:1.6.0-SNAPSHOT "docker-entrypoint.s…" 41 minutes ago Up 41 minutes 80/tcp, 443/tcp auth 0b7b9956da40 localstack/localstack:3.2.0 "docker-entrypoint.sh" 41 minutes ago Up 40 minutes (healthy) 127.0.0.1:4510-4559->4510-4559/tcp, 127.0.0.1:4566->4566/tcp, 5678/tcp localstack 2ab832b2ec61 postgres:14-alpine "docker-entrypoint.s…" 41 minutes ago Up 41 minutes 5432/tcp pass-docker-postgres-1 1b9fb04ddb02 dspace/dspace-postgres-pgcrypto:dspace-7.6 "docker-entrypoint.s…" 41 minutes ago Up 41 minutes 0.0.0.0:5432->5432/tcp dspacedb 6e8230eb7814 ghcr.io/eclipse-pass/pass-ui:1.6.0-SNAPSHOT "/bin/entrypoint.sh" 41 minutes ago Up 41 minutes 80/tcp pass-ui 84b402f8255c ghcr.io/eclipse-pass/demo-ldap:1.6.0-SNAPSHOT "./entrypoint.sh" 41 minutes ago Up 41 minutes 389/tcp ldap 1510d03d1f08 dspace/dspace-angular:dspace-7.6 "docker-entrypoint.s…" 41 minutes ago Up 41 minutes 0.0.0.0:4000->4000/tcp, 0.0.0.0:9876->9876/tcp dspace-angular 24e1ee4ac934 atmoz/sftp "/entrypoint pmcsftp…" 41 minutes ago Up 41 minutes 0.0.0.0:2222->22/tcp pmc-sftp-server 4253aa6924a0 ghcr.io/eclipse-pass/proxy:1.6.0-SNAPSHOT "entrypoint.sh" 41 minutes ago Up 40 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp proxy ``` {% endcode %} Get the container ID that is associated with `atmoz/sftp.`Using the container ID enter the SFTP container. {% code title="Get the Container ID" %} ``` docker ps -aqf name=pmc-sftp-server ``` {% endcode %} Copy the result from the "Get Container ID" command. Then run the "Enter the SFTP Container" command by replacing the "PasteContainerIDHere" with the previously copied container ID. {% code title="Enter the SFTP Container" %} ``` docker exec -it PasteContainerIDHere /bin/bash ``` {% endcode %} Now that you're in the SFTP server, navigate to the deposit. **Note:** the last directory that is a date will be the date when the deposit is made. ``` cd /home/pmcsftpuser/upload/2024-04-02 ``` And you will see all the deposits that you've made to NIHMS: ``` root@24e1ee4ac934:/home/pmcsftpuser/upload/2024-04-02# ls nihms-native-2022-05_2024-04-02_19-04-00_68.zip nihms-native-2022-05_2024-04-02_19-04-48_68.zip nihms-native-2022-05_2024-04-02_19-04-57_68.zip nihms-native-2022-05_2024-04-02_19-04-58_68.zip ``` Congratulations! You've simulated a manuscript deposit to the institutional repository (JScholarship) and NIHMS! ## Configuration All the defaults in the docker env configuration files will run without any modification, but if you need to change a port, URLs, or other configurations you can do so using the `.eclipse-pass.local_env` file. It's recommend while testing to keep these values the defaults, but if you need to change a port for a specific reason you can do so by modifiying the `.eclipse-pass.local_env`environmental variable file. ## Troubleshooting 1. **Docker Compose Fails to Start the Services** **Problem**: Users might encounter errors when running the `docker compose up` command due to various reasons such as network issues, Docker daemon not running, or insufficient permissions. **Solution**: Ensure Docker is running on your machine. Check your internet connection and firewall settings. If Docker is running and no networking connections issues are present, then trying updating or installing the [latest version of Docker](https://www.docker.com/get-started/). *** 2. **Insufficient System Memory Error** **Problem:** The application might fail or perform poorly if the system does not meet the minimum memory requirement. **Solution:** Close unnecessary applications to free up memory. Consider increasing your system's memory if persistent issues occur. Ensure you have at least 16GB of system memory available as recommended. If you're running Docker on a Windows machine, ensure that WSL2 and Docker Compose V2 are enabled in the settings. *** 3. **Unable to Access PASS on the web browser** **Problem:** After running the Docker compose command, the PASS application does not load or displays an error in the web browser. **Solution:** Verify that the containers are running by running the `docker ps` command. When running without deposit services you should see the following containers running(**Note**: image version may differ as it will be updated in the future e.g. 1.6.0-SNAPSHOT): ``` CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 813972bad937 ghcr.io/eclipse-pass/idp:1.6.0-SNAPSHOT "./entrypoint.sh" About a minute ago Up 56 seconds 4443/tcp, 8443/tcp idp 2aafc2de282e ghcr.io/eclipse-pass/pass-core-main:1.6.0-SNAPSHOT "./entrypoint.sh" About a minute ago Up 43 seconds (healthy) pass-core 960fbbed7458 ghcr.io/eclipse-pass/pass-auth:1.6.0-SNAPSHOT "docker-entrypoint.s…" About a minute ago Up 57 seconds 80/tcp, 443/tcp auth db263a61122d localstack/localstack:3.2.0 "docker-entrypoint.sh" About a minute ago Up 44 seconds (healthy) 127.0.0.1:4510-4559->4510-4559/tcp, 127.0.0.1:4566->4566/tcp, 5678/tcp localstack 663871b8a92b postgres:14-alpine "docker-entrypoint.s…" About a minute ago Up 58 seconds 5432/tcp pass-docker-postgres-1 b12319e6ccbb ghcr.io/eclipse-pass/pass-ui:1.6.0-SNAPSHOT "/bin/entrypoint.sh" About a minute ago Up 58 seconds 80/tcp pass-ui a51102277cff ghcr.io/eclipse-pass/demo-ldap:1.6.0-SNAPSHOT "./entrypoint.sh" About a minute ago Up 58 seconds 389/tcp ldap 576ff2aae621 ghcr.io/eclipse-pass/proxy:1.6.0-SNAPSHOT "entrypoint.sh" About a minute ago Up 54 seconds 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp proxy ``` When running with Deposit-Services and DSpace, as specified in the [advanced setup](#advanced-setup), you will see the following containers running: ``` CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 8359f7400413 ghcr.io/eclipse-pass/idp:1.6.0-SNAPSHOT "./entrypoint.sh" 41 seconds ago Up 33 seconds 4443/tcp, 8443/tcp idp 0d0abd03c480 dspace/dspace-solr:dspace-7.6 "/bin/bash -c 'init-…" 41 seconds ago Up 26 seconds 0.0.0.0:8983->8983/tcp dspacesolr b14227cec259 ghcr.io/eclipse-pass/deposit-services-core:1.6.0-SNAPSHOT "./entrypoint.sh" 41 seconds ago Up 9 seconds pass-deposit-services 238a6183e1ab ghcr.io/eclipse-pass/pass-core-main:1.6.0-SNAPSHOT "./entrypoint.sh" 41 seconds ago Up 12 seconds (health: starting) pass-core a7a24772c12d dspace/dspace:dspace-7.6-test "/bin/bash -c 'while…" 41 seconds ago Up 29 seconds 8000/tcp, 8009/tcp, 0.0.0.0:8080->8080/tcp dspace bc3ea5a2ec20 ghcr.io/eclipse-pass/pass-auth:1.6.0-SNAPSHOT "docker-entrypoint.s…" 43 seconds ago Up 35 seconds 80/tcp, 443/tcp auth a7403c2b34a6 localstack/localstack:3.2.0 "docker-entrypoint.sh" 43 seconds ago Up 15 seconds (healthy) 127.0.0.1:4510-4559->4510-4559/tcp, 127.0.0.1:4566->4566/tcp, 5678/tcp localstack 135dc909b2c1 postgres:14-alpine "docker-entrypoint.s…" 43 seconds ago Up 35 seconds 5432/tcp pass-docker-postgres-1 f8c24a38d438 ghcr.io/eclipse-pass/proxy:1.6.0-SNAPSHOT "entrypoint.sh" 43 seconds ago Up 29 seconds 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp proxy d9bb301597f4 ghcr.io/eclipse-pass/demo-ldap:1.6.0-SNAPSHOT "./entrypoint.sh" 43 seconds ago Up 37 seconds 389/tcp ldap 340435bf059f ghcr.io/eclipse-pass/pass-ui:1.6.0-SNAPSHOT "/bin/entrypoint.sh" 43 seconds ago Up 35 seconds 80/tcp pass-ui e6137acc145a dspace/dspace-angular:dspace-7.6 "docker-entrypoint.s…" 43 seconds ago Up 32 seconds 0.0.0.0:4000->4000/tcp, 0.0.0.0:9876->9876/tcp dspace-angular 778c5cec16d7 atmoz/sftp "/entrypoint pmcsftp…" 43 seconds ago Up 33 seconds 0.0.0.0:2222->22/tcp pmc-sftp-server 66daca298f22 dspace/dspace-postgres-pgcrypto:dspace-7.6 "docker-entrypoint.s…" 43 seconds ago Up 34 seconds 0.0.0.0:5432->5432/tcp dspacedb ``` If some of the containers are not running try `docker compose -p pass-docker down -v` and restart the containers by running the docker compose command mentioned in the [simple ](#simple-setup)or[ advanced setup](#advanced-setup) sections. # Collaboration with Other Institutions PASS is a community oriented Open Source Project and we're always looking to collaborate with other institutions and individuals that believe in [Open Access](https://en.wikipedia.org/wiki/Open_access). The PASS team has engaged in extensive collaborative efforts with various academic institutions, hosting providers, and government agencies to advance the development and integration of the PASS system. These collaborative initiatives were structured around two central themes: community engagement and communications. ### Community Engagement In the realm of community development, the PASS team, alongside members from the Eclipse Foundation, organized bi-weekly meetings during the 2023 Fall semester. These meetings fostered a collaborative environment with academic partners and hosting providers interested in the PASS system. Participants included organizations such as [CalTech](https://www.caltech.edu/), the [University of Virginia](https://www.virginia.edu/), the [University of Oregon](https://www.uoregon.edu/), the [University of Louisville](https://louisville.edu/), and the [National Center for Atmospheric Research (NCAR)](https://ncar.ucar.edu/), along with potential hosting providers [Lyrasis ](https://www.lyrasis.org/Pages/Main.aspx)and [TIND](https://www.tind.io/ir). The primary focus of these gatherings was to understand the challenges PASS could address, define software improvements for a PASS pilot, and better understand the deployment environments. This collaborative effort culminated in identifying pilot opportunities for 2024 and pinpointing institutional uses cases and integration requirements. ### Communications The PASS project team also undertook a campaign to spread awareness and foster discussions about PASS with a wide array of groups and institutions. Notable engagements included discussions on PASS integration with the National Institutes of Health (NIH) and the National Science Foundation (NSF), presentations to the HELIOS Infrastructure Working Group and the Big Ten Academic Alliance, and participation in various spotlight series and conferences. These communications extended throughout the year, touching base with organizations such as SPARC, the US Repository Network, Figshare, and the Invest in Open Infrastructure initiative, culminating in a presentation at the [Fall 2023 Coalition for Networked Information (CNI)](/pass-documentation-dev/welcome-guide/pass-demonstrations-conferences#cni-presentation-december-2023) and discussions with the NASA/CERN open science working group in early 2024. ### Looking Forward As we move forward into 2024, the PASS team is focused on facilitating the technical trials within the PASS community. These tasks are critical for enhancing the PASS system's functionality and ensuring its integration into broader academic and research infrastructures. The PASS team aims to enhance the PASS software by importing grant data and associated user data, developing a simple administrative user interface, adding additional user authentication options, and establishing a community testing environment. In addition there are two focused collaborations: 1. **Institutional Repository Deposit Integrations:** In collaboration with CalTech, integrate PASS with the repository [InvenioRDM](https://inveniordm.docs.cern.ch/). 2. **Integrate with a Funder Repository:** The team plans to engage with NSF PAR and DOE OSTI, with support from other interested groups such as Stanford, NCAR, and Dryad. This integration is pivotal for broadening the PASS ecosystem and enhancing its utility for various stakeholders. These strategic directions underscore our commitment to evolving PASS into a more robust, versatile platform that meets the needs of the academic and research communities. As we embark on these tasks, our focus remains on collaboration, innovation, and the continued pursuit of excellence in supporting open science and research endeavors. If you or your institution is interested in collaborating with the PASS Team, please send an email to our community [Google Group](mailto:pass-general@googlegroups.com). # Contributing to PASS ### Getting Involved Whether you are interested in simply improving the existing code base, or maybe even thinking about forking the project for use at your institution, it's a good idea to have a look at a running test instance and exercise the workflows. The easiest way to do this is to simply visit our [demo instance](https://demo.eclipse-pass.org/). ### Reporting a Bug If you've run across a bug in the Eclipse PASS application, letting us know about it is an important and an easy way to contribute. First, check to see if there's an existing issue for what you're observing by performing a search in the [issues list](https://github.com/eclipse-pass/main/issues). If you've discovered a new bug, create a new issue to tell us about it. You will need to create a [GitHub ](https://github.com/)account to do this, if you don't have one already. When creating an issue, please provide a concise description of the potential problem, the steps needed to reproduce it, and a description of how the behavior differs from what you need or expect. Also, provide details about your environment like OS, Java version, and Maven version, etc. ### Contributing Code or Documentation There is always a need for fixing bugs and we are continually looking to add new features. Contributions furthering these efforts are always welcome. It may be that you simply notice something that doesn't seem to work quite right or the list of issues on our [SonarQube Cloud](https://sonarcloud.io/organizations/eclipse-pass). You can check the GitHub [issues ](https://github.com/eclipse-pass/main/issues)to see if an issue has already been opened. If an issue was already opened, you may also comment on the issue if you have new information which might be helpful. If there is no existing issue, you might consider creating one ([see above](#reporting-a-bug)). The same process holds for exploring the addition of a new feature or updating documentation. To contribute to the code base, you will need an account on GitHub. Further, since PASS is an Eclipse Foundation project, contributors will need to [create an Eclipse account](https://accounts.eclipse.org/) and sign a [contributor agreement](https://www.eclipse.org/legal/ECA.php). To keep things simple, the email address you use to sign up for the Eclipse account should be the same as the one you use for your GitHub account. To make changes to code or documentation, you'll start by forking the GitHub repository you'd like to change into your own GitHub account. Once you have your own copy of the repository, create a Git branch for your work. Please name the branch based on the ID of the ticket associated with the issue, for example: `903-fix-bug-in-code`. You'll then be able to make changes, perform Git commits, and push those changes to your copy of the repository. Once your work is done, you'll need to create a Pull Request to let us know about your changes. [You'll find a more complete overview of this process here](https://opensource.com/article/19/7/create-pull-request-github). ### More Info More documentation about the PASS project can be found in the [Developer Documentation](/pass-documentation-dev/developer-documentation) and the [PASS Infrastructure](/pass-documentation-dev/infrastructure-documenation) sections of this documentation repository. General information about contributing to Eclipse projects can be found in the [Eclipse handbook](https://www.eclipse.org/projects/handbook/#contributing-contributors). # Community The PASS Community Documentation provides guidelines and resources for contributing to the Eclipse PASS project,\ covering key areas such as [developer guidelines](/pass-documentation-dev/community/developer-guidelines), the [project roadmap](/pass-documentation-dev/community/pass-roadmap), and\ ways to get involved. It outlines best practices for reporting bugs, contributing code, and testing to ensure a\ high-quality and collaborative development process. The documentation also includes the PASS roadmap, which details\ planned initiatives around user engagement, community building, system administration, and application enhancements. ## Getting Involved PASS is an open-source community project, and there are many ways to contribute—whether by providing feedback, reporting\ bugs, or writing code. Community members are encouraged to: * Review the existing code base and documentation. * Test the [PASS Docker](/pass-documentation-dev/developer-documentation/pass-docker) local test instance to understand workflows and features. * Join discussions on [Slack](https://eclipse-pass.slack.com/archives/C035MNLRD44) to ask questions or offer suggestions. * Follow the [pull request workflow](/pass-documentation-dev/community/developer-guidelines#pull-request-workflow) to submit code contributions. If you’re new to the project, a great starting point is to try the [PASS Docker](/pass-documentation-dev/developer-documentation/pass-docker)\ local test instance to explore the application and understand its core functionality. For more detailed developer information, visit the [PASS Developer Documentation](/pass-documentation-dev/developer-documentation) and read\ over the [developer guidelines](/pass-documentation-dev/community/developer-guidelines). To review the PASS roadmap and stay updated on future\ developments, see our [PASS Roadmap](/pass-documentation-dev/community/pass-roadmap). # Developer Guidelines The PASS Developer Guidelines provide detailed instructions for contributing to the PASS project, covering areas such as communication channels, testing procedures, pull request workflows, and documentation standards. It outlines expectations for contributors, the process for reporting issues, and best practices for maintaining code quality and project integrity. ## Getting Involved * Welcome to the PASS community! There are many ways to participate: trying out the PASS software, letting us know about bugs, suggesting documentation updates, or contributing code. After you’ve read this guide, if you have questions, please send us a message on our [Google Group](https://groups.google.com/g/pass-general), and we will be in touch shortly! * Looking for a place to start contributing? Our [SonarQube Cloud](https://sonarcloud.io/organizations/eclipse-pass) has a list of bugs/issues. * We primarily use Slack to communicate about PASS development. To be invited to our Slack workspace, please send us a message on our [Google Group](https://groups.google.com/g/pass-general). * Contributing to the project begins as a [contributor](https://www.eclipse.org/projects/handbook/#contributing-contributors) and may lead to being a [committer](https://www.eclipse.org/projects/handbook/#roles-cm). Whether you are a `contributor` or `committer`, you will need to sign up for an [Eclipse account](https://accounts.eclipse.org/user/login). * A `contributor` can add to and improve PASS by creating issues and submitting pull requests. You’ll find more information about both of these tasks below. * A `committer` is an individual who once was a `contributor`, but made significant contributions and was elected by the core team to become a `committer`. They can work directly in the repositories, create and close issues, and merge pull requests. ## Change Request/Bug Report Would you like to suggest a change to PASS or report a bug? This is done by submitting a GitHub issue. * When creating an issue to report a bug or suggest a new feature, use the [eclipse-pass/main repository](https://github.com/eclipse-pass/main/issues). * If available for your particular issue use one of the available [issue templates in the main repository](https://github.com/eclipse-pass/main/issues/new/choose). * If a suitable template doesn’t exist, use the default `Standard Issue` template. * Add a label if possible, if the label doesn’t exist, or you’re unsure of which one to use, send a message to the team on the Slack `#pass-dev` channel. * If possible, suggest a priority. If you’re unsure, leave it blank, and the team will determine the appropriate priority. ## Testing In general, we recommend the following procedures for testing: * If you're planning to submit code through a Pull Request (PR), please run tests locally first. For Java code, this can be done using `mvn verify` or `mvn clean install`. To test the complete project, [run pass docker](/pass-documentation-dev/welcome-guide/setup-run-pass) to test. * If you're planning to submit code which includes new tests, Martin Fowler’s [The Practical Testing Pyramid](https://martinfowler.com/articles/practical-test-pyramid.html) is a great resource for understanding how to structure tests. Additionally, we use this [definition for ITs](https://www.geeksforgeeks.org/software-engineering-integration-testing/) along with [Martin Fowler's definition](https://martinfowler.com/bliki/IntegrationTest.html). PASS has three different types of tests that are run against the application, and they are defined as: * **Unit Tests**: Unit tests focus on a single unit of code. They test very specific conditions, inputs, and expected outputs, validating that the unit behaves as intended. They are narrow in scope, and all other collaborators (e.g. other classes that are called by your class under test) are substituted with mocks or stubs. Unit tests alone do not guarantee the application as a whole will work as intended. * **Integration Tests**: They test the integration of your application with other parts that are not part of your application e.g. databases, external REST APIs. They are not as narrow as Unit Tests, but still test one integration point at a time. In addition, ITs focus on verifying the interactions and data exchange between different components or modules of a software application. * **Acceptance Tests**: They are a final validation step, ensuring PASS fits the workflow requirements for users. The [PASS Acceptance Tests](https://github.com/eclipse-pass/pass-acceptance-testing) runs through workflows using [Test Cafe](https://testcafe.io/) against an instance of PASS. All these tests must pass in order for PASS to be considered production ready. ### Back-end * **Unit Tests** * If introducing new functionality, please ensure new code is covered by at least one unit test which includes both success and failure states. * A few examples of this are [here](https://github.com/eclipse-pass/pass-support/blob/79ad19ed4d2592c342e7cdfdf652a8f7aef3eaa2/pass-deposit-services/deposit-core/src/test/java/org/eclipse/pass/deposit/service/DepositProcessorIT.java#L54) and [here](https://github.com/eclipse-pass/pass-core/blob/e9e853ac7eea05f595fdcd5342ddea99c0798e38/pass-core-main/src/test/java/org/eclipse/pass/object/ElidePassClientTest.java#L84). * If performing a bug fix, include a test to ensure that the bug was fixed. * In general, unit tests should run quickly. * **Integration Tests** * We recommend running integration tests in a test environment that mimics the production environment as closely as possible. * When adding or updating integration tests, please avoid making network requests to 3rd parties. * If needed, use test containers, wiremock, or mockbean. * Integration tests should be as fast as possible. * **Acceptance Tests** * Acceptance tests are used in PASS to verify correct functionality based on user requirements, so these should work correctly from a user’s perspective. * Updated whenever there are changes to user requirements, significant changes are made to the application, or when they break. * Automated so they can be run frequently and consistently. ### UI * When testing the UI it is helpful to run [Ember locally for faster iteration](https://github.com/eclipse-pass/main/blob/main/docs/dev/running-pass-ui-on-your-host-machine.md). * Include a unit test when you can, such as when functions don't interact with rendering. Otherwise, utilize component integration or ember application/acceptance tests where rendering is involved - this is what ember is best at. * [Pass-ui](https://github.com/eclipse-pass/pass-ui) is heavy on integration/application tests because it's rendering heavy and much of the business logic is in the back end. * If you write an encapsulated piece of UI like a component, that component should have at least 1 integration test. * Application level testing is done with mocked data using Mirage. This needs to be updated diligently, so it doesn't fall out of sync with the real back end. If you are updating the API in a way that changes the contract with pass-ui, please create an issue for updating the UI mocking to accommodate these changes. * At least one test should be added for bug fixes to prevent regression. * The [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing) acceptance tests are run frequently and can aid in making sure the UI application testing mocked responses are appropriate. * Ember provides a set of very helpful libraries to assist in testing UI components. It is best practice to use these helpers rather than inventing your own where possible: * [Ember test-helpers](https://github.com/emberjs/ember-test-helpers/blob/master/API.md) * [QUnit-dom](https://github.com/mainmatter/qunit-dom/blob/master/API.md) ## Commits * As a team, we do not enforce rigid rules about commit messages. However, we strive to write good commit messages by following these [guidelines by Chris Beams](https://cbea.ms/git-commit/). ## Documentation * We encourage you to read through the [PASS documentation style guide](https://docs.google.com/document/d/11aCooQCNhEq34yG9mGuFynY4xMDRJtPRuOOou9LaCgU/edit?usp=sharing) prior to submitting a pull request. * The PASS team uses [GitBook](https://www.gitbook.com/) for managing and creating documentation. There are two ways to create new documentation with this system, through the GitBook web interface and through our GitHub `pass-documentation` repository. * The process for creating, editing, and managing documentation will vary depending on which system you use: * GitHub: * Use a personal branch that is checked out from `development` and is rebased back into development. * Ensure the branch is up-to-date with `development` before creating a pull request. * Follow the same pull request guidelines mentioned in our Pull Request Workflow section. * GitBook: * Request to be added to the GitBook team. * Follow the change request process as outlined by the GitBook docs. * The same pull request guidelines mentioned in the Pull Request Workflow section apply here as well, such as who should be the reviewer. * **NOTE**: It is easy to merge directly from GitBook, ensure that the button at the top right is changed from `merge` to `request a review`. * Each repository in the PASS project should have a top level README. The following guidelines for these readme should include: * Overview of project purpose. * Links to appropriate sections in GitBook. * README in the main repo should include more details and a longer overview of the PASS project. ## Pull Request Workflow * If you are in the `contributor` role, you must first fork the repository you wish to make changes in and then submit the pull request to the upstream. If you’re not familiar with pull requests, see the [GitHub pull request documentation](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests). * Pull requests should be reviewed by at least one `committer` that is not the PR author before merging. * For pull requests created by `committers`, the pull request author is expected to perform the pull request merge after another committer has approved the pull request. * For pull requests created by contributors, the committer reviewing the pull request is responsible for merging the pull request after approval. * Ensure your branch is created from the latest version of `main`. * Branch names should include a ticket number and short description. * Example: `978-fix-nihms-loader-etl` * The description in your pull request should include the following: * Summary of major changes and what the pull request will accomplish. * Instructions identifying how to test the changes. * Ensure that every PR is linked to a relevant ticket. * Update or add new documentation to the `pass-documentation` repository. * This would be a separate pull request, see the Documentation section for this process. * A pull request should not be merged unless all automated checks pass. * The [SonarQube quality gates](https://github.com/eclipse-pass/pass-documentation/blob/development/infrastructure-documenation/code-quality-analysis/sonar-qube.md) are optional, but it is encouraged to address these code quality checks. * Merges should happen using the rebase strategy. * If there have been changes to the `main` code branch, you may want to rebase your branch on `main` for additional safety. * After a successful merge, delete the branch. ## Pull Request Review Process * When reviewing the code, here are some advised areas to consider: * Verify that the implemented logic aligns with the requirements and specifications. * Look for any potential bugs or logical errors. * Ensure code is adequately commented where necessary. * Verify that sensitive configurations are externalized and not hard-coded. * Ensure REST endpoints follow standard conventions (e.g., proper use of [HTTP methods](https://developer.mozilla.org/en-US/docs/Web/HTTP)). * Review the [SonarQube quality gates](https://github.com/eclipse-pass/pass-documentation/blob/development/infrastructure-documenation/code-quality-analysis/sonar-qube.md) and [JaCoCo code coverage](https://github.com/eclipse-pass/pass-documentation/blob/development/infrastructure-documenation/code-quality-analysis/jacoco.md) reports. * Review any unit/integration tests and ensure that they provide proper coverage. * Ensure proper use of mocks and stubs to isolate components during testing. * Identify and suggest refactoring for any [code smells](https://linearb.io/blog/what-is-a-code-smell). * Look for areas that could benefit from improved [design patterns or structures](https://www.baeldung.com/design-patterns-series). * Ensure that dependencies are properly managed and up-to-date. * Check for any potential conflicts or unused dependencies. * Review commit messages. * Build the project and run tests locally. ## Closing Issues * If an issue is linked to a pull request it will auto-close, however for issues that are not linked to a pull request, the committer performing the merge should close the completed issues. * Write up the final outcome in the issue and include this as a comment when closing the ticket. * Link any collaboration or design documents in the issue before closing. * Ensure all related PRs are linked and closed. * We recommend that the changes are deployed and tested in a staging or preproduction environment prior to completion. ## Protecting Sensitive Information * Take precaution to ensure that you’re not committing any credentials/keys/secrets. * If any sensitive information is accidentally committed, immediately notify the team on the `#pass-dev` Slack channel. * The core team will triage the severity of the leak and take appropriate actions. This may include removing it from the GitHub and GitBook commit history and rotating the compromised credentials. * Update GitHub secret scanning to catch any sensitive information that bypassed the original scan. # PASS Roadmap ## Eclipse PASS Project Roadmap This roadmap defines the primary initiatives of the Eclipse PASS Project, organized by anticipated release. If you are interested in helping to define and/or contribute to this roadmap, please reach out to us on our [Google Group](https://groups.google.com/g/pass-general). We're always happy to welcome new contributors! ### Priorities and goals #### User Engagement * Engage with current and potential PASS users to understand their needs and how PASS can better address their challenges. #### Community Engagement * Engage with and build out the broader community of institutions that have interest in utilizing PASS. * Provide a baseline method for other institutions to import the necessary data into PASS. * Prepare PASS to be deployed, used, and implemented at other institutions. * Add at least one additional IR deposit target. #### System Administration * Provide improved tooling and support for administering the PASS system. * Improve deployment infrastructure. * System monitoring and alerting. * Design and mock up an administrative dashboard that visualizes submissions and provides key statistics of the PASS application. #### Application Improvements * Integrate with at least one additional funder repository. * Improve code workflow using Continuous Integration and Continuous Delivery / Deployment. * Ensure the NIH integration is solid. * Work on technical debt and ensure system components are kept up-to-date. * Security and testing. #### Documentation * Improve documentation to prepare for collaboration with external institutions. # Release Notes ### Release v2.5.1 #### Date: April 13, 2026 Release Manager: Russ Poetker, JHU This release includes updates to resolve Critical CVEs in third-party dependencies. Additionally, Ember was upgraded to the latest LTS. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/40?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/2.5.1) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/2.5.1) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/2.5.1) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/2.5.1) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/2.5.1) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/2.5.1) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/2.5.1) ### Release v2.5.0 #### Date: March 30, 2026 Release Manager: Russ Poetker, JHU This is a maintenance release for PASS. Frontend and Backend dependencies were upgraded. In the pass-ui project, Ember was upgraded to the latest LTS and converted to Typescript. The Journal Data Loader was updated to support the new PMC file format. A bug related to DOI lookup error handling was fixed. Additionally, GitHub Actions workflows were updated to use immutable versions for third-party actions. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/39?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/2.5.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/2.5.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/2.5.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/2.5.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/2.5.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/2.5.0) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/2.5.0) ### Release v2.4.0 #### Date: November 18, 2025 Release Manager: Russ Poetker, JHU This release includes a change to the Deposit Services DSpace integration to support DSpace 9.x. There is also a change in the UI messaging related to PMC submission terminology. Additionally, the children POMs were cleaned up to remove inherited attributes. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/38?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/2.4.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/2.4.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/2.4.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/2.4.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/2.4.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/2.4.0) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/2.4.0) ### Release v2.3.0 #### Date: September 23, 2025 Release Manager: Russ Poetker, JHU This release includes two areas of change. The first is a set of changes to improve messaging to users describing what happens after the completion of a submission. Now an appropriate message in the UI will describe the next steps depending on the submission's target repositories. There were also several updates made to the grant selection tables with regard to improving UX and fixing a few bugs. The second area of change is in the grant loader where a new integration with the Fibi Grant Management System has been added. The PASS release workflow has also migrated to Central Portal for publishing artifacts to Maven central. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/36?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/2.3.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/2.3.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/2.3.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/2.3.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/2.3.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/2.3.0) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/2.3.0) ### Release v2.2.0 #### Date: May 28, 2025 Release Manager: Jared Galanis, JHU This release includes a refactoring of the way metadata schemas are processed and handled in pass-ui. This was a broad refactor that improved and simplified metadata state management and processing in pass-ui, replaced an outdated library (alpaca.js) in pass-ui that previously handled metadata forms with a more modern JavaScript library (survey.js), and moved the metadata schema service from pass-core into pass-ui. This release also removed support for the pass demo site, updating documentation and shutting down AWS resources that served the demo site. This release additionally resolved intermittent failures in pass-acceptance-testing CI runs and improved the documentation around the release process itself. During this release cycle the team also investigated and supported the migration from Sonatype OSSRH (being deprecated at the end of June) to Central Portal for publishing artifacts to Maven central. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/35?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/2.2.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/2.2.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/2.2.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/2.2.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/2.2.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/2.2.0) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/2.2.0) ### Release v2.1.1 #### Date: May 6, 2025 Release Manager: Mark Patton, JHU This is a patch release to fix a bug viewing the submission details page and fix a bug that prevented submissions with an embargo to DSpace from working. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/37?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/2.1.1) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/2.1.1) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/2.1.1) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/2.1.1) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/2.1.1) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/2.1.1) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/2.1.1) ### Release v2.1.0 #### Date: April 30, 2025 Release Manager: Jared Galanis, JHU This release introduces support in pass-core for Spring Cloud AWS to connect to S3 for reading config files. It also addresses technical debt in pass-support, includes test cleanup, and removes the MD5 checksum option in deposit services. Additionally, it resolves a page reload bug and a submit confirmation bug in pass-ui, along with refactoring the logic for submission file state management. Finally, documentation for SonarQube and JaCoCo has been added in this release. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/34?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/2.1.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/2.1.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/2.1.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/2.1.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/2.1.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/2.1.0) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/2.1.0) ### Release v2.0.0 #### Date: March 27, 2025 Release Manager: Tim Sanders, JHU This is a major release of PASS, marking the transition from the 1.x series to 2.x. It includes important security updates, new tools for improving code quality, and removed SWORD deposits. A research spike was also conducted to explore removing AlpacaJS. We had our first pull request from an external contributor with the [Pass-Docker InvenioRDM update](https://github.com/eclipse-pass/pass-docker/pull/389)! Highlights * Removed SWORD Support: SWORD deposit support has been removed. * Security Fixes: Updated dependencies identified with CVEs, improving the security posture of the PASS user interface. Strengthened Content Security Policy (CSP) headers to mitigate risks such as cross-site scripting (XSS) and content injection attacks. * Code Coverage Setup: Configured SonarQube to ingest JaCoCo reports, providing real-time code coverage status checks for pull requests. * Remove Alpaca Analysis: Investigated the feasibility of removing AlpacaJS from the metadata step, exploring potential improvements and simplifications for dynamic form generation. Additional Changes * Bug Fix: Addressed an issue with the proxy search dialog displaying at the top of the page rather than in its intended dialog. * Cleanup Documentation: Archived out-dated documentation. * Pass-Docker InvenioRDM: Update InvenioRDM to v12 in local docker environment. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/32?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/2.0.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/2.0.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/2.0.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/2.0.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/2.0.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/2.0.0) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/2.0.0) ### Release v1.15.0 #### Date: February 27, 2025 Release Manager: Mark Patton, JHU This release adds support for transforming JATS XML abstracts imported from a DOI to HTML for better display in a repository. Dependencies have been updated to address security problems found by an analysis of SBOMs. Fixes were made handling cleanup of test deposits which use the DSpace API. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/30?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.15.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.15.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.15.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.15.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.15.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.15.0) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/1.15.0) ### Release v1.14.1 #### Date: February 12, 2025 Release Manager: Russ Poetker, JHU This release is a patch release to fix a bug in notification services that was not allowing emails to be sent. This release also contains the new DSpace API transport in deposit services which is the new way to perform deposits into DSpace. The DSpace API transport will eventually replace the SWORD transport for DSpace deposits. Additionally, pass-docker was changed so that test PKI files are generated as needed on startup for the test IDP and InvenioRDM containers. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/31?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.14.1) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.14.1) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.14.1) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.14.1) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.14.1) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.14.1) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/1.14.1) ### Release v1.14.0 #### Date: January 29, 2025 Release Manager: Russ Poetker, JHU This release adds SonarQube Cloud for pass-core, pass-support, and pass-ui for Static Code Analysis to improve code quality and security (badges have been added to each project showing SonarQube Quality Gate status). The pass-core and pass-support projects now produce JaCoCo code coverage metrics. In a near-term future release, code coverage reports will be generated on SonarQube Cloud. The failed deposit retry rule has changed so that a failed deposit is only automatically retried if the target repository was unreachable. The release workflow has been updated to release the pass-documentation project. Various code cleanups and dependency updates have been made to the project. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/29?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.14.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.14.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.14.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.14.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.14.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.14.0) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/1.14.0) ### Release v1.13.0 #### Date: December 9, 2024 Release Manager: Mark Patton, JHU This release added SBOM creation to our release process. GitHub repository documentation now points to the new documentation site. Various code cleanups and dependency updates have been made to the Java backend. In addition work has started on support for direct deposit with the DSpace REST API. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/28?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.13.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.13.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.13.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.13.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.13.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.13.0) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/1.13.0) ### Release v1.12.0 #### Date: October 31, 2024 Release Manager: Tim Sanders, JHU With this version, we’ve implemented improvements in security, documentation, and fixed some minor bugs. The DOI service now avoids returning entries that lack a URL, and it provides more reliable functionality for retrieving filenames from URLs. Security enhancements include a review of security alerts, the removal of stack trace outputs on the 404 page, and the elimination of default values from sensitive application properties. Additionally, a new AWS feature allows application properties to be optionally loaded from the AWS SSM Parameter Store. Our documentation revamp is complete and accessible on our [new documentation site](https://docs.eclipse-pass.org)! [Tickets Completed](https://github.com/eclipse-pass/main/milestone/27?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.12.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.12.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.12.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.12.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.12.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.12.0) * [pass-documentation](https://github.com/eclipse-pass/pass-documentation/releases/tag/1.12.0) ### Release v1.11.0 #### Date: September 30, 2024 Release Manager: Mark Patton, JHU This release fixed a few small bugs around retrieving and displaying manuscripts associated with a DOI, added the ability to display a failure message from a repository to a user and refreshed the Shibboleth setup of the local development environment. Work on enhancing our documentation and moving it to Gitbook is ongoing. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/26?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.11.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.11.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.11.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.11.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.11.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.11.0) ### Release v1.10.0 #### Date: August 28, 2024 Release Manager: Russ Poetker, JHU This release focused on a new Deposit Services repository integration, NIHMS Data Loader automation, and Release process and Documentation improvements. Deposit services has been enhanced to be able to deposit into InvenioRDM. This can also be tested locally with pass-docker being able to start a local instance of InvenioRDM. There is a new automation available in NIHMS Data Loader for refreshing the NIHMS API Token. We made a change to the pass-core/pass-support releases to align the maven repackage plugin configuration with its latest recommendations. As part of this change, the repackaged jar file is no longer deployed to Maven Central during release. The team continued work on overhauling and improving the existing documentation. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/24?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.10.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.10.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.10.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.10.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.10.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.10.0) ### Release v1.9.1 #### Date: August 1, 2024 Release Manager: Russ Poetker, JHU This release fixes a bug with the layout of some of the pages in the UI. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/25?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.9.1) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.9.1) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.9.1) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.9.1) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.9.1) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.9.1) ### Release v1.9.0 #### Date: July 31, 2024 Release Manager: Russ Poetker, JHU This release focused on improving deployment tests, addressing technical debt, and improving documentation. Deployment tests were updated to make deposits into downstream repositories optional. If a deployment test deposit is made into a downstream DSpace repository, it will be automatically deleted after the test completes. More deprecations in pass-ui were fixed which allowed pass-ui to be upgraded to Ember v5.8. The pass-ui module now uses Embroider and pnpm for building. The team continued work on overhauling the existing documentation. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/23?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.9.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.9.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.9.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.9.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.9.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.9.0) ### Release v1.8.0 #### Date: July 1, 2024 Release Manager: Jared Galanis, JHU This release focused on improving documentation, addressing technical debt, and fixing bugs. CSRF protection was added to several PASS components. An optional InvenioRDM instance was added to pass-docker. Many deprecations that were preventing an upgrade of pass-ui to Ember v5.x were addressed. The IDP configuration was reworked to be loaded more dynamically via a url instead of a file. The team advanced an overhaul of existing documentation, where many older sources of documentation were reviewed for accuracy, relevance and categorization, and some of which were subsequently synthesized into new forms of documentation. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/22?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.8.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.8.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.8.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.8.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.8.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.8.0) ### Release v1.7.0 #### Date: May 30, 2024 Release Manager: Russ Poetker, JHU This release focused on adding a GitHub workflow that will complete the release of all PASS components. The pass-core metadata schema service was changed as a first step in supporting InvenioRDM integration. There were several documentation tasks completed such as a Review Manual, Style Guide, and first round of reviews. There has been steps made to eventually add IaC for PASS. OpenTofu has been selected as the IaC tool; developing the terraform modules is in progress. A few smaller bugs were also fixed. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/21?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.7.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.7.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.7.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.7.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.7.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.7.0) ### Release v1.6.1 #### Date: May 7, 2024 Release Manager: Mark Patton, JHU This release fixes a bug which may prevent login and tweaks timeouts for monitoring NIHMS email. Tickets Completed: * [930](https://github.com/eclipse-pass/main/issues/930) * [974](https://github.com/eclipse-pass/main/issues/974) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.6.1) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.6.1) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.6.1) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.6.1) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.6.1) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.6.1) ### Release v1.6.0 #### Date: April 30, 2024 Release Manager: Mark Patton, JHU This release focused on simplifying authentication support, writing documentation to make collaboration easier, and automated testing against a live PASS instance. The pass-core component took over the responsibility for authentication and mediating access to pass-ui resources. This allowed us to remove the no longer needed pass-auth component. The deposit services now also cleanup after the new automated tests. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/20?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.6.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.6.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.6.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.6.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.6.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.6.0) ### Release v1.5.0 #### Date: March 28, 2024 Release Manager: Timothy Sanders, JHU This release focused on use cases for the planned Admin UI, documentation for the PASS Welcome Guide, and enhancements to the grant and nihms data loaders. We added updated parameters to the nihms loader for scheduled environments, and revised the nihms email processing. The grant loader had updates to its aggregation rules, CSV ingest, and extended test coverage. We simplified our CI/CD pipeline by creating a single action to deploy all PASS components to a specified environment. Began consolidating the authentication process to integrate Spring Security into pass-core, enhancing flexibility and security. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/19?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.5.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.5.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.5.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.5.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.5.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/1.5.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.5.0) ### Release v1.4.0 #### Date: February 28, 2024 Release Manager: Russ Poetker, JHU This release focused on updating dependency versions and enforcing clean dependency management in the PASS backend repositories. The required configuration architecture was simplified for the nihms and grant data loaders by making these Spring applications. We began working on a new documentation repository supported by GitBook, more to come on this in the near future. We improved the file delete action on the UI by deleting such files from the backend. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/18?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.4.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.4.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.4.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.4.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.4.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/1.4.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.4.0) ### Release v1.3.0 #### Date: January 31, 2024 Release Manager: Mark Patton, JHU This release focused on updating the interactions with NIHMS. A service was added to handle email messages from NIHMS about submission status. We made GitHub actions for Java snapshot and release builds consistent and more robust. We switched the grant loader to a CSV format which will make it easier to import grant data from other systems. For the UI, we improved the accessibility of the UI and the interaction with external links in the workflow. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/17?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.3.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.3.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.3.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.3.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.3.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/1.3.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.3.0) ### Release v1.2.0 #### Date: November 30, 2023 Release Manager: Timothy Sanders, JHU This release focused on increasing the security of PASS and making interactions with external services more robust, notably the interface with NIHMS. Parameterized queries were added to the grant loader to enhance security. We've made substantial upgrades to the NIHMS data transfer within our Deposit Services and enhancements to the NIHMS loader. Updates were made to the data model documentation and client-side pagination support has been added in the UI. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/16?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.2.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.2.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.2.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.2.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.2.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/1.2.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.2.0) ### Release v1.1.0 #### Date: October 26, 2023 Release Manager: Mark Patton, JHU This release focused on getting PASS ready for deployment in production. We did a great deal of testing of the user interface, backend services, and interactions with repositories. We found and fixed a large number of bugs. We significantly improved the performance of grant loading. We made accessibility improvements to the user interface. In addition, we added support for depositing to repositories without requiring a journal be entered by the user. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/15?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.1.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.1.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.1.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.1.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.1.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/1.1.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.1.0) ### Release v1.0.0 #### Date: September 29, 2023 Release Manager: Jared Galanis, JHU This release focused on setting up a PASS for production readiness. We resolved a large number of bugs in the user interface and the API / backend services. We added optimistic locking to Submission and Deposit entities to ensure more expected behavior when users edit a shared resource. We also did work on tooling for data migration and remediation. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/12?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/1.0.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/1.0.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/1.0.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/1.0.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/1.0.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/1.0.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/1.0.0) ### Release v0.9.0 #### Date: August 30, 2023 Release Manager: John Abrahams, JHU This release focused on setting up a staging environment for the PASS application. We deployed PASS to the new environment, integrated single sign-on and the data loaders, and fixed a number of bugs that were discovered in the refactored codebase. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/13?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/0.9.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/0.9.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/0.9.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/0.9.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/0.9.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/0.9.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/0.9.0) ### Release v0.8.0 #### Date: July 28, 2023 Release Manager: Mark Patton, JHU This release introduces updated Java implementations of pass-deposit-services. All of the major functionality of PASS has now been ported to the new framework. In addition more testing was added to the pass-core file service and support for the file service was added to pass-data-client. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/11?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/0.8.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/0.8.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/0.8.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/0.8.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/0.8.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/0.8.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/0.8.0) Release Manager: Mark Patton, JHU ### Release v0.7.0 #### Date: June 29, 2023 Release Managers: Christopher Shannon, JHU and Russell Poetker, JHU This release introduces updated java implementations of the pass-nihms-loader and pass-notification-services projects. This release also introduces support for sending submission and deposit JMS message from pass-core, adds access control to the file-service, removes pass-ui-public from pass-docker, and cleans up the SAML configuration in pass-auth. [Tickets Completed](https://github.com/eclipse-pass/main/milestone/10?closed=1) Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/0.7.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/0.7.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/0.7.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/0.7.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/0.7.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/0.7.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/0.7.0) ### Release v0.6.0 #### Date: May 31, 2023 Release Manager: Jared Galanis, JHU This release introduces java implementations of the pass-journal-loader, pass-grant-loader and submission status service. This release also introduces support for user token authentication, updates to use of Java 17 in several repositories, converts pass-auth to TypeScript, integrates the user interface with the API for the policy service, introduces a simplified branding strategy along with default branding fallbacks to enable organization specific look and feel, and provides an action for publishing to an AWS SNS (Simple Notification Service) topic to facilitate deploying to AWS infrastructure. Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/0.6.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/0.6.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/0.6.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/0.6.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/0.6.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/0.6.0) * [pass-ui-public](https://github.com/eclipse-pass/pass-ui-public/releases/tag/0.6.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/0.6.0) ### Release v0.5.0 #### Date: April 27, 2023 Release Manager: Timothy Sanders, JHU This release introduces the Metadata Schema Service and the Policy Service API. The Metadata Schema Service provides JSON schemas for repository metadata requirements. The Policy Service API determines the policies applicable to a given Submission, as well as the repositories that a Submission must be deposited into. Release Automation has been expanded to include [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing) and [pass-docker](https://github.com/eclipse-pass/pass-docker). Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/0.5.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/0.5.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/0.5.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/0.5.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/0.5.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/0.5.0) * [pass-ui-public](https://github.com/eclipse-pass/pass-ui-public/releases/tag/0.5.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/0.5.0) ### Release v0.4.0 #### Date: March 30, 2023 Release Managers: John Abrahams, JHU and Christopher Shannon, JHU This release introduces a new user service and access control. The release also upgraded ember to the latest LTS Ember 4. * Updated Ember packages and 3rd party dependencies * Fixed styling post Ember 4 upgrade * Introduces User Service Integration * Introduces Access Control * Standardized the entrypoint of dockerfiles to point at an entrypoint.sh file Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/0.4.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/0.4.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/0.4.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/0.4.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/0.4.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/0.4.0) * [pass-ui-public](https://github.com/eclipse-pass/pass-ui-public/releases/tag/0.4.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/0.4.0) ### Release v0.3.0 #### Date: February 28, 2023 Release Manager: John Abrahams, JHU This release introduces a new file handling service for dealing with file uploads in PASS. Releases are now largely automated using GitHub workflows. * Release automations using GitHub workflows. Snapshot versions are published automatically and releases can be triggered manually in the GitHub UI * Add file API to pass-core for handling file related create, read, and delete operations * Add service to pass-core to look for publicly available manuscripts for a given DOI Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/0.3.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/0.3.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/0.3.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/0.3.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/0.3.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/0.3.0) * [pass-ui-public](https://github.com/eclipse-pass/pass-ui-public/releases/tag/0.3.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/0.3.0) ### Release v0.2.0 #### Date: January 18, 2023 Release Manager: Jim Martino, JHU Release 0.2.0 provides a major upgrade to the backend architecture of the PASS application. The Fedora Repository has been replaced with a completely new REST API built using Elide and backed by Postgres. This change allows the PASS API to be tailored more directly to the purposes of the PASS application, provides considerable performance enhancements, and reduces maintenance burden. The structure of the projects making up the PASS application have also been streamlined to simplify release and deployment procedures. These changes require updates to be made across the application, such as replacing all uses of the Fedora API within PASS with calls to the new API. For 0.2.0, this work is completed sufficiently to provide a demonstration of PASS application capabilities, but certain parts of the application are currently mocked. Full functionality will be restored in a future release. Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/0.2.0) * [pass-core](https://github.com/eclipse-pass/pass-core/releases/tag/0.2.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/0.2.0) * [pass-acceptance-testing](https://github.com/eclipse-pass/pass-acceptance-testing/releases/tag/0.2.0) * [pass-support](https://github.com/eclipse-pass/pass-support/releases/tag/0.2.0) * [pass-auth](https://github.com/eclipse-pass/pass-auth/releases/tag/v0.2.0) * [pass-ui-public](https://github.com/eclipse-pass/pass-ui-public/releases/tag/v0.2.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/0.2.0) Some major changes for v0.2.0 are: * Replacement of Fedora storage with Elide / postgres. * Creation of a pass-core repository which contains REST services previously in separate projects/images * Elimination of functionality written in Go in previous releases. These have either been implemented in Java or eliminated * Refactoring of authorization functionality in javascript closer to the UI ### Release v0.1.0 #### Date: August 3, 2022 Release Manager: John Abrahams, JHU This is the initial release of the Eclipse PASS codebase. The following changes were made to the code after completing the transition to Eclipse: Naming changes - updating code to transition to the Eclipse PASS name Version alignment - ensuring all project components utilize a consistent versioning scheme Data model alignment - ensuring all project components utilize the same version of the data model Adjustments to release process - updates to allow release of Java components to Maven Central with a new groupId Introduction of code style guide - ensuring code conforms to consistent guidelines Adjustments to testing methods - transitioning to the use of GitHub Actions for the execution of unit and integration testing Initial documentation - providing a starting point for development and deployment documentation Release Components: * [main](https://github.com/eclipse-pass/main/releases/tag/0.1.0) * [pass-ui-public](https://github.com/eclipse-pass/pass-ui-public/releases/tag/v0.1.0) * [pass-ui](https://github.com/eclipse-pass/pass-ui/releases/tag/v0.1.0) * [pass-ember-adapter](https://github.com/eclipse-pass/pass-ember-adapter/releases/tag/v0.1.0) * [pass-java-client](https://github.com/eclipse-pass/pass-java-client/releases/tag/0.1.0) * [pass-authz](https://github.com/eclipse-pass/pass-authz/releases/tag/0.1.0) * [pass-deposit-services](https://github.com/eclipse-pass/pass-deposit-services/releases/tag/0.1.0) * [pass-messaging-support](https://github.com/eclipse-pass/pass-messaging-support/releases/tag/0.1.0) * [pass-notification-services](https://github.com/eclipse-pass/pass-notification-services/releases/tag/0.1.0) * [pass-package-providers](https://github.com/eclipse-pass/pass-package-providers/releases/tag/0.1.0) * [pass-doi-service](https://github.com/eclipse-pass/pass-doi-service/releases/tag/0.1.0) * [pass-download-service](https://github.com/eclipse-pass/pass-download-service/releases/tag/0.1.0) * [pass-policy-service](https://github.com/eclipse-pass/pass-policy-service/releases/tag/0.1.0) * [pass-indexer-checker](https://github.com/eclipse-pass/pass-indexer-checker/releases/tag/0.1.0) * [pass-indexer](https://github.com/eclipse-pass/pass-indexer/releases/tag/0.1.0) * [pass-journal-loader](https://github.com/eclipse-pass/pass-journal-loader/releases/tag/0.1.0) * [pass-nihms-loader](https://github.com/eclipse-pass/pass-nihms-loader/releases/tag/0.1.0) * [pass-grant-loader](https://github.com/eclipse-pass/pass-grant-loader/releases/tag/0.1.0) * [pass-fcrepo-jsonld](https://github.com/eclipse-pass/pass-fcrepo-jsonld/releases/tag/0.1.1) * [pass-fcrepo-jms](https://github.com/eclipse-pass/pass-fcrepo-jms/releases/tag/0.1.0) * [pass-docker](https://github.com/eclipse-pass/pass-docker/releases/tag/0.1.0) These repositories were involved in the release, but do not have a `0.1.0` release: * [pass-fcrepo-module-auth-rbacl](https://github.com/eclipse-pass/pass-fcrepo-module-auth-rbacl) * [modeshape](https://github.com/eclipse-pass/modeshape) # Developer Documentation The PASS Developer Documentation encompasses all aspects of PASS required to understand and contribute to the PASS\ development. The main areas of PASS Development are PASS Core, PASS UI, Data Loaders, Deposit Services,\ Notification Services, and Acceptance Tests. All of these components work together to bring together a comprehensive\ system to disseminate research to their proper repositories. Each of these modules has a distinct role, such as managing\ data ingestion, handling user interactions, automating data transformations, and making deposits to their intended\ downstream repository. In all, these components interact to create an efficient workflow that supports research\ dissemination in various institutional and federal compliance scenarios. The Data Loaders handle data ingestion from external systems like PubMed Central, NIH, FIBI\ (JHU Grant Management System), transforming grant, journal, and manuscript submission data into a standardized format\ within PASS. Deposit Services then manage the packaging and transfer of these submissions to downstream repositories\ such as Pubmed Central and institutional repositories. Notification Services provide alerts and updates to relevant\ stakeholders based on submission workflows and events. This modular structure, built primarily using Java and Spring\ Boot, allows for flexibility and extensibility in accommodating different institutional requirements and third-party\ integrations. The developer documentation also details the technical setup, configuration, and deployment of each PASS component,\ including environment-specific configurations, database setup, authentication management, and integration with cloud\ services provided by AWS. PASS utilizes RESTful APIs for data access and manipulation, while its microservices\ communicate asynchronously using message queues to support scalable and distributed deployments. By following a modular\ and loosely-coupled architecture, PASS is able to evolve rapidly, incorporating new compliance requirements and\ enhancing repository deposit capabilities across academic and research institutions. **Table of Contents:** 1. [Use cases](/pass-documentation-dev/developer-documentation/use-cases) 2. [PASS Core](/pass-documentation-dev/developer-documentation/pass-core) 3. [PASS UI](/pass-documentation-dev/developer-documentation/pass-ui) 4. [Metadata Schema](https://github.com/eclipse-pass/pass-documentation/blob/development/developer-documentation/metadata-schema.md) 5. [Data Loaders](/pass-documentation-dev/developer-documentation/data-loaders) 6. [Deposit Services](/pass-documentation-dev/developer-documentation/deposit-service) 7. [Notification Services](/pass-documentation-dev/developer-documentation/notification-service) 8. [Pass Acceptance Testing](/pass-documentation-dev/developer-documentation/pass-acceptance-testing) 9. [PASS Docker](/pass-documentation-dev/developer-documentation/pass-docker) 10. [Release](/pass-documentation-dev/developer-documentation/release) --- [Next Page](/llms-full.txt/1)