Enabling S3 Uploads For Fork PR Test Reports

Hey guys! Today, we're diving into an interesting challenge: how to enable the upload of test reports to S3 from fork-based pull request Continuous Integration (CI) runs. This is super important for projects like ROCm, where contributions come from both internal team members and external contributors. Currently, our CI setup only uploads artifacts and test reports to S3 for runs triggered from internal branches. This means when someone opens a pull request from a fork, the CI runs smoothly, but the test reports and logs don't make their way to our S3 artifact buckets. This makes it tough to review test results for contributions from outside the organization, and we want to fix that!

The Problem: Missing Test Reports from Forked PRs

So, let's break down the problem a bit more. Imagine you're an external contributor, you've made some awesome changes, and you've submitted a pull request. The CI runs, everything looks good, but the core team can't easily see your test results because they're not being uploaded to S3. This creates a bottleneck in the review process. We need a way to ensure that test logs and reports from fork-based PRs are also uploaded to S3, making it easier for everyone to collaborate and maintain code quality.

To reiterate, the current CI workflows are configured to upload build artifacts and test reports to S3 only when runs are triggered from internal branches. This includes branches like main, develop, or any feature branches within the ROCm organization. However, when a pull request originates from a fork—an external repository—the CI run, while completing successfully, doesn't copy the crucial test reports or logs to the designated S3 artifact buckets. These buckets are typically named something like therock-artifacts or therock-artifacts-external. This discrepancy makes it significantly harder to review and assess the contributions made via fork-based pull requests. The absence of these reports hinders the ability to thoroughly evaluate the quality and impact of external contributions.

This issue directly impacts the efficiency of the code review process and the overall maintainability of the project. Without access to test results, reviewers have to rely on other means to verify the correctness and stability of the changes, which can be time-consuming and less reliable. Furthermore, it can discourage external contributions if the process of getting a pull request reviewed and merged is perceived as cumbersome due to the lack of readily available test data. Therefore, addressing this issue is vital for fostering a healthy and collaborative open-source development environment. We aim to create a seamless experience for all contributors, regardless of their organizational affiliation, by ensuring that all necessary testing artifacts are accessible for review.

Goal: Securely Enable S3 Uploads for All PRs

The main goal here is to enable the upload of test logs and reports to S3 for pull requests coming from forks. But, and this is a big but, we need to do this while keeping things secure. We can't just give everyone write access to our S3 buckets! We need to prevent untrusted forks from gaining direct access to our internal S3 resources. So, it's a balancing act between making things accessible and keeping things safe. Our primary goal is to enable the upload of test logs and reports to S3 for pull requests originating from forks, but this must be achieved while preserving the security of our internal resources.

One of the critical aspects of this goal is preventing untrusted forks from gaining direct write access to our internal S3 buckets. If we were to simply allow uploads from any fork, we would open ourselves up to potential security risks. Malicious actors could potentially exploit this access to upload harmful content, overwrite existing data, or otherwise compromise our infrastructure. Therefore, it's imperative that any solution we implement includes robust security measures to mitigate these risks. The challenge lies in striking a balance between accessibility and security. We want to make it as easy as possible for contributors to submit their work and for reviewers to assess it, but we can't compromise the integrity of our systems in the process.

To achieve this, we need to carefully consider the mechanisms we use to authenticate and authorize uploads. We need to ensure that only legitimate CI jobs are allowed to upload data, and that the data is stored in the correct location with the appropriate permissions. This requires a solution that is both technically sound and adheres to best practices for security and access control. By addressing these concerns, we can create a secure and efficient workflow for managing contributions from both internal and external sources. The end result will be a more collaborative and productive development environment, where contributions are easily reviewed and integrated, and where the security of our infrastructure is never compromised.

Proposed Solutions: OIDC and Conditional Uploads

To tackle this, we're looking at two main approaches:

1. Use OIDC (OpenID Connect): Imagine giving the test runner a temporary keycard that only works for a specific task and a specific location. That's kind of what OIDC does. It allows us to grant temporary, scoped access for the test runner to upload artifacts to designated S3 buckets. This access is restricted by the repository context and the event type, so it's super secure.
2. Implement Conditional Upload Logic: This is like having a smart gatekeeper in the workflow. The workflow checks where the pull request is coming from (internal branch or external fork) and then decides the correct S3 bucket to upload to. This way, we can ensure that artifacts from forks go to a specific bucket, maybe one with stricter access controls.

OIDC for Secure Temporary Access

Let's dive a bit deeper into the OIDC approach. OpenID Connect (OIDC) is an authentication layer built on top of the OAuth 2.0 protocol. It allows us to verify the identity of the CI job and grant it temporary access to S3 based on that identity. Think of OIDC as a way to provide a short-lived, highly specific key that unlocks only the necessary resources. This is a much more secure approach than, say, using long-lived access keys, which could be compromised and used for malicious purposes.

With OIDC, we can configure our CI provider (like GitHub Actions) to act as an OIDC Identity Provider. When a CI job runs, it can request a JSON Web Token (JWT) from the provider. This JWT contains claims about the job, such as the repository it's running in, the branch it's running on, and the event that triggered it (e.g., a pull request). We can then configure our AWS Identity and Access Management (IAM) policies to trust these JWTs and grant temporary access to S3 based on the claims they contain. This allows us to create fine-grained access control policies that limit the scope of access to only what is necessary for the job to complete.

For example, we can create an IAM policy that allows a CI job running in the ROCm/rccl repository, triggered by a pull request from a fork, to upload artifacts to a specific S3 bucket designated for fork-based PRs. This policy would only be in effect for the duration of the job, and it would not grant access to any other resources. This approach significantly reduces the risk of unauthorized access and data breaches. By leveraging OIDC, we can ensure that only authorized CI jobs can upload artifacts to S3, and that they can only upload to the buckets they are explicitly granted access to.

Conditional Upload Logic for Dynamic Bucket Selection

The second proposed solution involves implementing conditional upload logic within the CI workflow itself. This approach is like having a smart traffic controller directing uploads to different destinations based on the origin of the pull request. The core idea is to analyze the context of the CI run and determine whether the pull request originates from an internal branch or an external fork. Based on this determination, the workflow will dynamically select the appropriate S3 bucket to upload the test reports and logs.

This conditional logic can be implemented using environment variables and conditional statements within the CI workflow configuration. For example, GitHub Actions provides environment variables that indicate the source repository and branch of the pull request. We can use these variables to create a condition that checks if the source repository is the same as the target repository (internal branch) or if it's a fork (external repository). If it's a fork, the workflow will upload the artifacts to a designated S3 bucket for fork-based PRs. If it's an internal branch, the workflow will upload to the standard internal S3 bucket.

This approach offers a flexible way to manage uploads from different sources. We can configure different buckets with different access control policies, ensuring that data from external forks is stored separately and securely. For instance, we might create a separate S3 bucket for fork-based PRs with stricter access controls and potentially shorter retention policies. This helps to isolate the data and minimize the risk of unauthorized access or accidental data leakage. Furthermore, the conditional logic can be easily adapted to accommodate changes in the project's contribution model or security requirements. By implementing this dynamic bucket selection, we can create a more robust and secure CI workflow for managing contributions from both internal and external sources.

Next Steps: Validating and Testing the Solutions

Okay, so we have a plan! Now, what's next? Here are the next steps we're going to take:

1. Define and Validate IAM/OIDC Policies: We need to create the rules that govern who can access what in S3. This means defining the IAM (Identity and Access Management) policies and OIDC configurations for fork-based workflow runs.
2. Update Workflow Logic: We'll need to tweak our CI workflows to support those conditional S3 upload paths. This involves adding the logic to determine the correct bucket based on the PR source.
3. Test, Test, Test: We'll thoroughly test uploads from both internal and fork-based PRs to make sure everything is working as expected. We want to ensure consistent behavior across the board.

Defining and Validating IAM/OIDC Policies

The first critical step in enabling secure artifact uploads from fork-based PRs is to meticulously define and validate the Identity and Access Management (IAM) and OpenID Connect (OIDC) policies. This process involves crafting policies that grant the necessary permissions for CI jobs to upload artifacts to S3, while simultaneously adhering to the principle of least privilege. In essence, we aim to provide the minimum level of access required for the job to function correctly, thereby minimizing the potential attack surface.

For OIDC, this entails configuring our CI provider (such as GitHub Actions) to act as an Identity Provider and establishing trust relationships with our AWS account. This involves defining the conditions under which a CI job can request a JWT (JSON Web Token) and how AWS will validate the authenticity and integrity of that token. We need to specify the allowed audience, the issuer, and the claims that the JWT must contain in order to be considered valid. These claims typically include information about the repository, branch, and event that triggered the CI job.

On the AWS side, we create IAM roles that trust the OIDC Identity Provider and grant specific permissions to access S3 buckets. These permissions should be narrowly scoped to allow only the necessary actions, such as uploading objects to a particular bucket and prefix. The IAM policy should also include conditions that further restrict access based on the claims in the JWT. For example, we might restrict access to a specific S3 bucket based on the repository and branch that triggered the CI job. This ensures that only authorized jobs can upload artifacts to the designated buckets.

Validating these policies is crucial to ensure that they function as intended and do not inadvertently grant excessive permissions. We can use tools like the AWS Policy Simulator to test the policies and verify that they allow the intended actions and deny unintended ones. Thorough testing and validation are essential to maintaining the security and integrity of our CI/CD pipeline.

Updating Workflow Logic for Conditional S3 Uploads

The next key step in this process is updating the CI workflow logic to incorporate conditional S3 upload paths. This involves modifying the workflow configuration to dynamically determine the correct S3 bucket to upload artifacts to, based on the origin of the pull request. This requires implementing logic that can distinguish between pull requests originating from internal branches and those originating from forks.

Most CI platforms, including GitHub Actions, provide environment variables that contain information about the source and target repositories of a pull request. We can leverage these environment variables to create conditional statements within the workflow configuration. For example, we can check if the source repository is the same as the target repository. If they are the same, it indicates that the pull request is from an internal branch, and the workflow should upload artifacts to the standard internal S3 bucket. If the source and target repositories are different, it indicates that the pull request is from a fork, and the workflow should upload artifacts to a designated S3 bucket for fork-based PRs.

The workflow logic can be implemented using a variety of scripting languages, such as Bash or Python, or using the built-in expression language provided by the CI platform. The key is to create a clear and concise set of rules that accurately determine the correct S3 bucket based on the pull request context. This may involve checking multiple environment variables and combining them using logical operators.

In addition to determining the correct bucket, the workflow logic may also need to handle other aspects of the upload process, such as setting the appropriate access control lists (ACLs) on the uploaded objects. For example, we might want to grant different permissions to objects uploaded from internal branches versus those uploaded from forks. This can be achieved by programmatically setting the ACLs during the upload process. By carefully updating the workflow logic, we can create a robust and flexible system for managing S3 uploads from both internal and external contributors.

Thorough Testing for Consistent Behavior

The final, and arguably most crucial, step in this process is thorough testing. We need to rigorously test the updated CI workflows to ensure that they function correctly in all scenarios. This means testing uploads from both internal branches and fork-based PRs, and verifying that the artifacts are being uploaded to the correct S3 buckets with the appropriate permissions.

Testing should encompass a variety of scenarios, including pull requests with different types of changes, different file sizes, and different levels of complexity. We should also test edge cases and potential failure scenarios, such as network interruptions or S3 outages. This helps to identify any weaknesses in the workflow logic and ensure that it can handle unexpected situations gracefully.

The testing process should involve both automated and manual testing. Automated tests can be used to verify the basic functionality of the workflow and to catch regressions. Manual testing is essential for verifying more complex scenarios and for ensuring that the workflow meets the needs of the users. This may involve manually creating pull requests, triggering CI runs, and verifying that the artifacts are uploaded correctly.

Consistent behavior is paramount. We need to ensure that the upload process is reliable and predictable, regardless of the source of the pull request or the conditions under which the CI job is running. Any inconsistencies or errors in the upload process can lead to delays in the code review process and can potentially compromise the security of our systems. Therefore, thorough testing is essential to building confidence in the updated CI workflows and ensuring that they meet our requirements for security, reliability, and performance.

Conclusion: Enabling Collaboration and Security

This issue is all about enabling secure artifact and test log uploads from fork-based PR runs. By doing this, we're making it easier for contributors outside the ROCm organization to share their test results for review. This is a big win for collaboration and helps us build a stronger, more vibrant community. Plus, by carefully considering security and implementing solutions like OIDC and conditional upload logic, we're ensuring that we can do this safely and responsibly. This initiative is crucial for fostering collaboration within the ROCm community, as it ensures that contributors outside the organization can seamlessly share their test results for review.

By enabling secure artifact and test log uploads from fork-based PR runs, we are significantly lowering the barrier to entry for external contributions. This allows contributors to showcase the quality of their work and makes it easier for reviewers to assess the impact of their changes. This streamlined process not only accelerates the code review cycle but also promotes a more inclusive and collaborative development environment.

Moreover, the solutions we are implementing, such as OIDC and conditional upload logic, are designed to ensure that this enhanced accessibility does not come at the expense of security. By carefully controlling access to our S3 buckets and implementing robust authentication mechanisms, we can confidently enable external contributions while safeguarding our internal resources. This proactive approach to security is essential for maintaining the integrity of our project and ensuring that it remains a trusted platform for innovation.

In conclusion, this effort to enable S3 uploads for fork-based PR test reports represents a significant step forward in our commitment to open collaboration and secure development practices. By embracing these principles, we are not only fostering a more vibrant community but also building a more resilient and trustworthy software ecosystem. We're excited to see the positive impact this will have on the ROCm project and the broader open-source community. Thanks for following along, and stay tuned for more updates as we move forward!