A modular Terraform implementation of the GenAI Intelligent Document Processing (IDP) Accelerator, designed to transform unstructured documents into structured data at scale using AWS's latest AI/ML services.
This project provides a Terraform-based implementation of the GenAI Intelligent Document Processing Accelerator, offering flexible deployment options and infrastructure-as-code management for Intelligent Document Processing workflows.
This solution is in experimental stage. While it implements security best practices, conduct thorough testing and security review before production use.
This repository uses a dual-version scheme to track both the upstream IDP solution and the Terraform implementation:
IDP_VERSION: Tracks the upstream GenAI IDP Accelerator versionVERSION: Tracks this Terraform implementation using the format{IDP_VERSION}-tf.{TERRAFORM_VERSION}
| VERSION | Meaning |
|---|---|
0.3.18-tf.0 |
IDP v0.3.18, initial Terraform implementation |
0.3.18-tf.1 |
IDP v0.3.18, first Terraform patch |
0.3.18-tf.2 |
IDP v0.3.18, second Terraform patch |
0.4.9-tf.0 |
IDP v0.4.9, initial Terraform implementation |
The Terraform version (tf.X) resets to 0 when the upstream IDP version changes.
- Modular Architecture: Reusable Terraform modules for flexible composition
- Multiple AI Processing Patterns: Three distinct approaches for different use cases
- Serverless Design: Built on AWS Lambda, Step Functions, and other serverless technologies
- Security First: KMS encryption, IAM least privilege, and VPC support with comprehensive security fixes
- Comprehensive Monitoring: Built-in CloudWatch dashboards and alerting
- Web Interface: Optional React-based UI for document management
- Knowledge Base: Query processed documents using natural language
- Document Discovery: Upload and analyze documents for pattern discovery
- Chat with Documents: Interactive Q&A with processed documents using foundation models
- Enhanced Permissions: Comprehensive IAM policies aligned with AWS security best practices
The GenAI IDP Accelerator supports three different processing patterns, each optimized for specific use cases:
- Best for: Standard document types with well-defined schemas
- Processing: Managed service with built-in document understanding
- Setup: Requires existing BDA project configuration
- Customization: Schema-based with predefined document classes
- Recent Updates: Enhanced permissions for DynamoDB, SSM Parameter Store, and Bedrock Data Automation APIs
- Best for: Custom document types requiring flexible extraction
- Processing: Multi-stage pipeline with foundation models (Claude, Nova)
- Setup: Fully automated deployment
- Customization: Complete control over classification and extraction logic
- Best for: Specialized document processing with custom models
- Processing: Fine-tuned UDOP models for classification + Bedrock for extraction
- Setup: Requires trained UDOP model artifacts
- Customization: Custom model training and deployment
genai-idp-terraform/
├── modules/ # Reusable Terraform modules
│ ├── processors/ # Document processing patterns
│ │ ├── bda-processor/ # Pattern 1: Bedrock Data Automation
│ │ ├── bedrock-llm-processor/ # Pattern 2: Bedrock LLM Processing
│ │ └── sagemaker-udop-processor/ # Pattern 3: SageMaker UDOP Processing
│ ├── processing-environment/ # Core processing infrastructure
│ ├── processing-environment-api/ # GraphQL API for status tracking
│ ├── web-ui/ # React-based web interface
│ ├── knowledge-base/ # Vector database for RAG
│ └── [supporting modules]/ # Additional infrastructure components
├── examples/ # Complete deployment examples
│ ├── bda-processor/ # BDA processor deployment
│ ├── bedrock-llm-processor/ # Bedrock LLM processor deployment
│ ├── sagemaker-udop-processor/ # SageMaker UDOP processor deployment
│ └── [other examples]/ # Additional example configurations
└── sources/ # Original CDK implementation (reference only)
Based on comprehensive security analysis and AWS best practices, the solution now includes:
- DynamoDB: Full CRUD operations for tracking and configuration tables
- SSM Parameter Store: Secure parameter management with GetParameter, PutParameter, and GetParametersByPath
- Bedrock Data Automation: Complete API access including GetDataAutomationProject, ListDataAutomationProjects, GetBlueprint, and GetBlueprintRecommendation
- S3 Enhanced Permissions: Full CRUD operations including DeleteObject and GetBucketLocation for comprehensive bucket management
- Modular HITL Policies: Separate conditional policies for Human-in-the-Loop functionality
- Optional BDA Metadata Table: Configurable access to BDAMetadataTable based on deployment requirements
- Environment Variable Consistency: Standardized CONFIGURATION_TABLE_NAME across all Lambda functions
- Least Privilege Access: IAM policies follow principle of least privilege with specific resource ARNs
- Conditional Resource Access: HITL functionality deployed only when required using Terraform for_each patterns
- Resource Separation: Clear separation between core processing permissions and optional HITL permissions
Before deploying the solution, ensure you have:
- Terraform: Version 1.0 or later
- AWS CLI: Configured with appropriate credentials
- Docker: For building Lambda deployment packages (if needed)
- AWS Account: With appropriate permissions for all services used
- Bedrock Model Access: Enable access to required models in the AWS Console
- Service Quotas: Ensure adequate quotas for Lambda, Step Functions, etc.
Important: Before deploying, you must enable access to Bedrock models:
- Go to the AWS Console
- Navigate to Amazon Bedrock
- Click "Model access" in the left navigation
- Request access to the models you plan to use:
- Claude 3 Sonnet:
anthropic.claude-3-sonnet-20240229-v1:0 - Claude 3 Haiku:
anthropic.claude-3-haiku-20240307-v1:0 - Nova Pro:
us.amazon.nova-pro-v1:0 - Titan Text Express:
amazon.titan-text-express-v1
- Claude 3 Sonnet:
- Click the checkbox next to each model and "Request model access"
This is a one-time manual step that cannot be automated through Terraform.
Select the pattern that best fits your use case:
# For flexible, custom document processing (Recommended for most use cases)
cd examples/bedrock-llm-processor
# For standard documents with existing BDA project
cd examples/bda-processor
# For specialized processing with custom UDOP models
cd examples/sagemaker-udop-processorCopy and customize the example configuration:
# Copy the example configuration
cp terraform.tfvars.example terraform.tfvars
# Edit with your specific values
nano terraform.tfvarsMinimal configuration example:
# terraform.tfvars
aws_region = "us-east-1"
prefix = "my-idp"
tags = {
Environment = "development"
Project = "document-processing"
}# Initialize Terraform
terraform init
# Review the deployment plan
terraform plan
# Deploy the infrastructure
terraform applyAfter deployment, Terraform will output important information:
# View deployment outputs
terraform output
# Example outputs:
# web_ui_url = "https://d1234567890.cloudfront.net"
# api_endpoint = "https://abcdef.appsync-api.us-east-1.amazonaws.com/graphql"
# input_bucket = "my-idp-input-documents-abc123"Once deployed, you can process documents in several ways:
- Navigate to the
web_ui_urlfrom your Terraform outputs - Upload documents through the drag-and-drop interface
- Monitor processing status in real-time
- View and download extraction results
# Upload a document to trigger processing
aws s3 cp my-document.pdf s3://your-input-bucket/import boto3
s3 = boto3.client('s3')
s3.upload_file('local-document.pdf', 'your-input-bucket', 'documents/document.pdf')# Check Step Functions executions
aws stepfunctions list-executions --state-machine-arn <state-machine-arn>
# View CloudWatch logs
aws logs describe-log-groups --log-group-name-prefix "/aws/lambda/your-prefix"# terraform.tfvars
aws_region = "us-east-1"
prefix = "my-idp"
# Optional: Custom model selection
classification_model_id = "anthropic.claude-3-sonnet-20240229-v1:0"
extraction_model_id = "anthropic.claude-3-sonnet-20240229-v1:0"
# Optional: Performance tuning
max_processing_concurrency = 50
classification_max_workers = 20# Enable additional features
enable_web_ui = true
enable_knowledge_base = true
enable_evaluation = true
enable_summarization = true
# Discovery Configuration (optional)
discovery = {
enabled = true
}
# Chat with Document Configuration (optional)
chat_with_document = {
enabled = true
# Optional: Add Bedrock Guardrail for content filtering
guardrail_id_and_version = "your-guardrail-id:1"
}
# HITL (Human-in-the-Loop) Configuration
# Set to non-null value to enable HITL functionality
bda_metadata_table_arn = "arn:aws:dynamodb:us-east-1:123456789012:table/my-bda-metadata-table"
# Custom configuration
custom_config = {
classification = {
method = "multimodalPageLevelClassification"
}
extraction = {
max_tokens = 8000
}
}┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Web UI │ │ GraphQL API │ │ S3 Buckets │
│ (CloudFront) │◄──►│ (AppSync) │◄──►│ (Input/Output) │
└─────────────────┘ └──────────────────┘ └─────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Step Functions Workflow │
├─────────────────┬─────────────────┬─────────────────────────────┤
│ OCR │ Classification │ Extraction │
│ (Textract) │ (Bedrock/UDOP) │ (Bedrock) │
└─────────────────┴─────────────────┴─────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Monitoring & Logging │
│ (CloudWatch Dashboards & Alarms) │
└─────────────────────────────────────────────────────────────────┘
- Least Privilege: All IAM policies follow the principle of least privilege
- Resource-Specific ARNs: Policies reference specific resource ARNs rather than wildcards
- Conditional Access: HITL functionality requires explicit configuration to enable
- KMS Encryption: All data encrypted at rest and in transit
- VPC Support: Optional VPC deployment for network isolation
- Secure Parameter Management: SSM Parameter Store for sensitive configuration
- CloudWatch Integration: Comprehensive logging and monitoring
- AWS Config: Resource compliance monitoring
- CloudTrail: API call auditing and tracking
Before deploying this solution to production, consider implementing the following AWS best practices. These recommendations help ensure security, reliability, and cost optimization.
- Lambda Concurrent Execution Limits: Configure reserved concurrency for each Lambda function to prevent runaway costs and protect downstream services. This solution currently has 43 Lambda functions without configured limits.
- Lambda Dead Letter Queue (DLQ): Configure DLQ for Lambda functions to capture and analyze failed asynchronous invocations. This solution currently has 37 Lambda functions without configured DLQ.
- Lambda VPC Configuration: Configure Lambda functions inside a VPC when accessing private resources. This solution currently has 10 Lambda functions without VPC configuration. Note: Only required for functions accessing VPC resources.
- Lambda Environment Variable Encryption: Configure Lambda functions to encrypt environment variables using KMS customer-managed keys. This solution currently has 39 Lambda functions without environment variable encryption.
- Lambda Code Signing: Configure Lambda functions to validate code signatures to ensure code integrity. This solution currently has 46 Lambda functions without code-signing validation.
- (Additional recommendations will be added here)
- CodeBuild Encryption with CMK: Configure CodeBuild projects to use customer-managed KMS keys for encrypting build artifacts, environment variables, and logs. This solution currently has 3 CodeBuild projects without CMK encryption.
- CloudWatch Log Group Encryption: Configure CloudWatch Log Groups to use KMS encryption for log data at rest. This solution currently has 7 CloudWatch Log Groups without KMS encryption.
- (Additional recommendations will be added here)
- CloudWatch Alarms: Set up alarms for critical metrics and error rates
- CloudWatch Log Retention: Configure CloudWatch Log Groups with appropriate retention periods (minimum 1 year for production). This solution currently has 37 CloudWatch Log Groups without adequate retention.
- (Additional recommendations will be added here)
- Resource Tagging: Implement comprehensive tagging strategy for cost allocation
- (Additional recommendations will be added here)
- Multi-AZ Deployment: Deploy critical components across multiple availability zones
- CloudFront Origin Failover: Configure CloudFront with origin groups for automatic failover to secondary origins. This solution currently has 1 CloudFront distribution without origin failover.
- CloudFront Geo Restriction: Configure geo restrictions based on compliance and security requirements. This solution currently has 1 CloudFront distribution without geo restriction configured.
- (Additional recommendations will be added here)
For detailed guidance on implementing these best practices, see the AWS Best Practices Documentation.
This project includes comprehensive documentation built with Material for MkDocs, covering everything from getting started to advanced deployment scenarios.
- GitHub Pages: https://awslabs.github.io/genai-idp-terraform/
- Auto-deployed: Updated automatically with each commit to main branch
- Mobile-friendly: Responsive design for all devices
Run the documentation locally for development:
# Quick start (automated setup)
cd docs && ./serve.sh
# Manual setup
cd docs
pip install -r requirements.txt
mkdocs serveVisit http://127.0.0.1:8000 to view the documentation.
- Getting Started: Prerequisites, installation, and quick start
- Terraform Modules: Detailed module documentation
- Examples: Real-world implementation examples
- Deployment Guides: Deployment best practices
- Security Guide: Security best practices and compliance
- FAQs: Common questions and troubleshooting
- Contributing: How to contribute to the project
- Live Examples: All examples reference real, working Terraform configurations
- Cost Estimates: Detailed cost breakdowns for each deployment pattern
- Architecture Diagrams: Visual representations of system components
- Step-by-step Guides: Detailed instructions for every deployment scenario
- Troubleshooting: Common issues and their solutions
- Best Practices: Recommended configurations and security guidelines
cd examples/bedrock-llm-processor
# Configure for financial documents (invoices, statements, forms)
# Uses Claude 3 Sonnet for high accuracycd examples/bda-processor
# Configure for standard legal documents
# Uses BDA for consistent, schema-based processing
# Enhanced with comprehensive IAM permissions and HITL supportcd examples/sagemaker-udop-processor
# Deploy with fine-tuned UDOP model
# Ideal for specialized document typesError: AccessDeniedException: You don't have access to the model
Solution: Enable model access in the Bedrock console (see Prerequisites)
Error: User is not authorized to perform: bedrock:InvokeModel
Solution: Ensure your AWS credentials have the required permissions
Error: LimitExceededException: Rate exceeded
Solution: Adjust max_processing_concurrency in your configuration
Error: Access denied to BDA metadata table
Solution: Ensure bda_metadata_table_arn is correctly configured if using HITL functionality
- Module Documentation: Each module has detailed README with troubleshooting
- AWS Documentation: Refer to service-specific AWS documentation
- CloudWatch Logs: Check function logs for detailed error information
- Security Documentation: Review security guide for IAM and permission issues
- GitHub Issues: Report bugs and request features
We welcome contributions! Each module includes detailed contributor documentation:
- Main Contributing Guide: Overall repository guidelines and workflow
- BDA Processor: Pattern 1 implementation guidance
- Bedrock LLM Processor: Pattern 2 implementation guidance
- SageMaker UDOP Processor: Pattern 3 implementation guidance
- Processing Environment: Core infrastructure guidance
- Web UI: Frontend development guidance
When contributing security-related changes:
- Follow AWS security best practices
- Use least privilege IAM policies
- Include comprehensive testing for permission changes
- Document security implications in pull requests
This project is licensed under the Apache License 2.0. See the LICENSE file for details.
- AWS CDK Version: CDK implementation
- Terraform Documentation: Terraform best practices
- AWS Bedrock Documentation: Bedrock service documentation
- Amazon Textract Documentation: OCR service documentation
- AWS Step Functions Documentation: Workflow orchestration
- AWS Security Best Practices: Security guidance and compliance
| Name | Version |
|---|---|
| terraform | >= 1.0 |
| aws | >= 5.0 |
| awscc | >= 0.70.0 |
| random | >= 3.1 |
| Name | Version |
|---|---|
| aws | 6.26.0 |
| random | 3.7.2 |
| Name | Source | Version |
|---|---|---|
| assets_bucket | ./modules/assets-bucket | n/a |
| bda_processor | ./modules/processors/bda-processor | n/a |
| bedrock_llm_processor | ./modules/processors/bedrock-llm-processor | n/a |
| human_review | ./modules/human-review | n/a |
| idp_common_layer | ./modules/idp-common-layer | n/a |
| processing_environment | ./modules/processing-environment | n/a |
| processing_environment_api | ./modules/processing-environment-api | n/a |
| processor_attachment | ./modules/processor-attachment | n/a |
| reporting | ./modules/reporting | n/a |
| sagemaker_udop_processor | ./modules/processors/sagemaker-udop-processor | n/a |
| user_identity | ./modules/user-identity | n/a |
| web_ui | ./modules/web-ui | n/a |
| Name | Type |
|---|---|
| aws_iam_role_policy.authenticated_user_permissions | resource |
| random_string.suffix | resource |
| aws_caller_identity.current | data source |
| aws_partition.current | data source |
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| bda_processor | Configuration for BDA processor | object({ |
null |
no |
| bedrock_llm_processor | Configuration for Bedrock LLM processor | object({ |
null |
no |
| data_tracking_retention_days | Document tracking data retention period in days | number |
365 |
no |
| deletion_protection | Enable deletion protection for Cognito resources | bool |
true |
no |
| enable_api | Enable GraphQL API for programmatic access and notifications | bool |
true |
no |
| enable_encryption | Whether encryption is enabled. Set to true when providing encryption_key_arn. This is needed to avoid Terraform plan-time unknown value issues. | bool |
true |
no |
| encryption_key_arn | ARN of the KMS key used for encrypting resources in the document processing workflow | string |
n/a | yes |
| evaluation | Configuration for document processing evaluation against baseline | object({ |
{ |
no |
| force_rebuild_layers | Force rebuild of Lambda layers regardless of requirements changes | bool |
false |
no |
| human_review | Configuration for human review functionality in document processing | object({ |
{ |
no |
| input_bucket_arn | ARN of the S3 bucket where source documents to be processed are stored | string |
n/a | yes |
| knowledge_base | Configuration for AWS Bedrock Knowledge Base functionality | object({ |
{ |
no |
| lambda_tracing_mode | X-Ray tracing mode for Lambda functions. Valid values: Active, PassThrough | string |
"Active" |
no |
| log_level | Log level for Lambda functions | string |
"INFO" |
no |
| log_retention_days | CloudWatch log retention period in days | number |
7 |
no |
| output_bucket_arn | ARN of the S3 bucket where processed documents and extraction results will be stored | string |
n/a | yes |
| prefix | Prefix for resource names | string |
"genai-idp" |
no |
| region | AWS region to deploy resources | string |
"us-east-1" |
no |
| reporting | Configuration for reporting and analytics functionality | object({ |
{ |
no |
| sagemaker_udop_processor | Configuration for SageMaker UDOP processor | object({ |
null |
no |
| tags | Tags to apply to all resources | map(string) |
{} |
no |
| user_identity | Configuration for external Cognito User Identity resources. If provided, the module will use this instead of creating its own user identity resources. | object({ |
null |
no |
| vpc_security_group_ids | List of security group IDs for Lambda functions (optional) | list(string) |
[] |
no |
| vpc_subnet_ids | List of subnet IDs for Lambda functions to run in (optional) | list(string) |
[] |
no |
| web_ui | Web UI configuration object | object({ |
{ |
no |
| working_bucket_arn | ARN of the S3 bucket for temporary working files during document processing | string |
n/a | yes |
| Name | Description |
|---|---|
| api | API resources (if enabled) |
| assets_bucket | Shared assets bucket information |
| name_prefix | Name prefix used for all resources |
| processing_environment | Processing environment resources |
| processor | Document processor details |
| processor_type | Type of document processor used |
| user_identity | User identity resources |
| web_ui | Web UI resources (if enabled) |