Skip to main content

Amazon MSK (Managed Streaming for Apache Kafka) - Interview Preparation

Total Questions: 290 Levels: Core Fundamentals (100), Intermediate (120), Advanced (80)


Table of Contents​

  1. Core Fundamentals
  2. Intermediate Topics
  3. Advanced Topics

1. Core Fundamentals​

Q1: What is Amazon MSK and how does it differ from self-managed Kafka?​

Technical Answer: Amazon MSK is a fully managed service for Apache Kafka that handles provisioning, configuration, and maintenance.

MSK vs Self-Managed:

FeatureSelf-Managed KafkaAmazon MSK
ProvisioningManualAutomated
PatchingManualAutomated
MonitoringBuild yourselfCloudWatch
ScalingManual rebalancingAuto-scaling (Serverless)
VPCYour responsibilityManaged
ZooKeeperManual managementManaged (or KRaft)

MSK Cluster Types:

import boto3

kafka = boto3.client('kafka')

# Provisioned cluster
response = kafka.create_cluster(
ClusterName='my-kafka-cluster',
KafkaVersion='3.5.1',
NumberOfBrokerNodes=3,
BrokerNodeGroupInfo={
'InstanceType': 'kafka.m5.large',
'ClientSubnets': ['subnet-123', 'subnet-456', 'subnet-789'],
'SecurityGroups': ['sg-123'],
'StorageInfo': {
'EbsStorageInfo': {'VolumeSize': 1000}
}
},
EncryptionInfo={
'EncryptionInTransit': {
'InCluster': True,
'ClientBroker': 'TLS'
}
},
ClientAuthentication={
'Sasl': {'Iam': {'Enabled': True}}
}
)

# MSK Serverless
response = kafka.create_cluster(
ClusterName='my-serverless-cluster',
Serverless={
'VpcConfigs': [{
'SubnetIds': ['subnet-123', 'subnet-456'],
'SecurityGroups': ['sg-123']
}]
},
ClientAuthentication={'Sasl': {'Iam': {'Enabled': True}}}
)

🔍 Interviewer Notes: Testing managed services understanding and Kafka architecture.

❌ Common Mistakes:

  • Choosing provisioned when serverless fits (or vice versa)
  • Not understanding broker vs topic vs partition
  • Wrong instance sizing

Q2: How do you produce and consume messages with MSK?​

Technical Answer: Use Kafka client libraries with IAM authentication.

Producer Example:

from kafka import KafkaProducer
import json
from aws_msk_iam_sasl_signer import MSKAuthTokenProvider

def get_auth_token():
return MSKAuthTokenProvider.generate_auth_token('us-east-1')

bootstrap_servers = [
'b-1.my-cluster.kafka.us-east-1.amazonaws.com:9098',
'b-2.my-cluster.kafka.us-east-1.amazonaws.com:9098',
'b-3.my-cluster.kafka.us-east-1.amazonaws.com:9098'
]

producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM',
sasl_aws_msk_iam_auth_token_provider=get_auth_token,
value_serializer=lambda v: json.dumps(v).encode('utf-8'),
key_serializer=lambda k: k.encode('utf-8') if k else None,
batch_size=16384,
linger_ms=10,
compression_type='gzip',
retries=3,
acks='all'
)

# Send message
future = producer.send(
topic='orders',
key='order-123',
value={'order_id': '123', 'amount': 99.99}
)
record_metadata = future.get(timeout=10)
producer.flush()
producer.close()

Consumer Example:

from kafka import KafkaConsumer
import json

consumer = KafkaConsumer(
'orders',
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM',
sasl_aws_msk_iam_auth_token_provider=get_auth_token,
group_id='order-processors',
auto_offset_reset='earliest',
enable_auto_commit=False,
max_poll_records=500,
value_deserializer=lambda v: json.loads(v.decode('utf-8'))
)

def process_messages():
try:
while True:
message_batch = consumer.poll(timeout_ms=1000)

for topic_partition, messages in message_batch.items():
for message in messages:
try:
process_order(message.value)
except Exception as e:
print(f"Failed: {e}")
break
else:
consumer.commit_async()
finally:
consumer.close()

🔍 Interviewer Notes: Testing Kafka client implementation and authentication.

❌ Common Mistakes:

  • Using auto-commit with at-least-once requirements
  • Not handling rebalancing
  • Wrong security protocol

Q3: How do you configure MSK topics and partitions?​

Technical Answer: Topics and partitions determine parallelism and throughput.

Topic Management:

from kafka.admin import KafkaAdminClient, NewTopic

admin_client = KafkaAdminClient(
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM',
sasl_aws_msk_iam_auth_token_provider=get_auth_token
)

# Create topic
topic = NewTopic(
name='high-throughput-events',
num_partitions=12,
replication_factor=3,
topic_configs={
'retention.ms': '604800000',
'cleanup.policy': 'delete',
'compression.type': 'producer',
'min.insync.replicas': '2',
'message.max.bytes': '1048576'
}
)

admin_client.create_topics([topic])

Partition Strategy:

# Same key -> Same partition -> Ordering guaranteed
producer.send('orders', key='customer-456'.encode(), value=order)

# No key = round-robin
producer.send('events', value=event)

# Custom partitioning
partition = hash(event['region']) % 12
producer.send('events', value=event, partition=partition)

🔍 Interviewer Notes: Testing Kafka architecture and partitioning strategies.

❌ Common Mistakes:

  • Too few partitions (can't scale consumers)
  • Too many partitions (rebalancing overhead)
  • Not considering ordering requirements

Q4: What is consumer lag and how do you handle it?​

Technical Answer: Consumer lag is the difference between the latest message and the last committed offset.

Monitoring Lag:

from kafka.admin import KafkaAdminClient

admin_client = KafkaAdminClient(...)

def check_consumer_lag(group_id):
consumer_groups = admin_client.list_consumer_groups()

for group in consumer_groups:
if group[0] == group_id:
offsets = admin_client.list_consumer_group_offsets(group_id)

for topic_partition, offset_metadata in offsets.items():
committed = offset_metadata.offset
end_offsets = admin_client.list_offsets({topic_partition: -1})
latest = end_offsets[topic_partition].offset

lag = latest - committed
print(f"{topic_partition}: lag={lag}")

if lag > 10000:
print("WARNING: High lag!")

Reducing Lag:

  • Increase consumer instances (max = partitions)
  • Increase max.poll.records
  • Optimize processing logic
  • Scale up broker instance types

🔍 Interviewer Notes: Testing production operations understanding.


Q5: How do you implement MSK Connect for data ingestion?​

Technical Answer: MSK Connect is managed Kafka Connect for streaming data between MSK and external systems.

S3 Sink Connector:

import boto3

kafkaconnect = boto3.client('kafkaconnect')

connector = kafkaconnect.create_connector(
connectorName='msk-to-s3',
kafkaCluster={
'apacheKafkaCluster': {
'bootstrapServers': ','.join(bootstrap_servers),
'vpc': {
'subnets': ['subnet-123', 'subnet-456'],
'securityGroups': ['sg-123']
}
}
},
capacity={
'autoScaling': {
'maxWorkerCount': 4,
'mcuCount': 1,
'minWorkerCount': 1
}
},
connectorConfiguration={
'connector.class': 'io.confluent.connect.s3.S3SinkConnector',
's3.bucket.name': 'my-kafka-backup',
's3.region': 'us-east-1',
'topics': 'orders,payments',
'format.class': 'io.confluent.connect.s3.format.json.JsonFormat',
'partitioner.class': 'io.confluent.connect.storage.partitioner.TimeBasedPartitioner',
'partition.duration.ms': '3600000',
'flush.size': '1000'
},
kafkaConnectVersion='2.7.1',
serviceExecutionRoleArn='arn:aws:iam::123456789:role/MSKConnectRole'
)

🔍 Interviewer Notes: Testing data pipeline architecture.


Q6: What are Kafka brokers, topics, and partitions?​

Technical Answer: These are the core abstractions in Kafka's architecture.

Definitions:

  • Broker: Kafka server that stores data and serves clients
  • Topic: Category/feed name to which records are published
  • Partition: Ordered, immutable sequence of records within a topic

Architecture:

┌─────────────────────────────────────────────────────────────┐
│ Kafka Cluster │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Broker 1 │ │ Broker 2 │ │ Broker 3 │ │
│ │ (Leader) │ │ (Follower) │ │ (Follower) │ │
│ └──────┬──────┘ └─────────────┘ └─────────────┘ │
│ │ │
│ Topic: orders │
│ ┌─────────────────────────────────────────────────┐ │
│ │ Partition 0 (Leader on Broker 1) │ │
│ │ [0:msg1] [1:msg2] [2:msg3] [3:msg4] ... │ │
│ └─────────────────────────────────────────────────┘ │
│ ┌─────────────────────────────────────────────────┐ │
│ │ Partition 1 (Leader on Broker 2) │ │
│ │ [0:msg5] [1:msg6] [2:msg7] ... │ │
│ └─────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

Key Points:

  • Each partition has one leader and multiple replicas
  • Producers write to the leader, consumers read from leader or followers
  • Partitions enable parallelism (more partitions = more throughput)

🔍 Interviewer Notes: Testing fundamental Kafka architecture understanding.

❌ Common Mistakes:

  • Not understanding leader/follower replication
  • Creating too few partitions (bottleneck)
  • Not knowing partition ordering guarantees

Q7: How does Kafka message retention work?​

Technical Answer: Kafka retains messages based on time and/or size policies, not consumption.

Retention Configuration:

from kafka.admin import KafkaAdminClient, ConfigResource, ConfigResourceType

admin_client = KafkaAdminClient(
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM'
)

# Configure topic retention
config_resources = [ConfigResource(
ConfigResourceType.TOPIC,
'orders',
configs={
'retention.ms': '604800000', # 7 days in milliseconds
'retention.bytes': '1073741824', # 1 GB per partition
'cleanup.policy': 'delete' # or 'compact' or 'compact,delete'
}
)]

admin_client.alter_configs(config_resources)

Retention Policies:

PolicyDescriptionUse Case
deleteRemove old messages after retention period/timeEvent streaming, logs
compactKeep latest value per key, tombstone for deletesState changes, KTables
compact,deleteBoth compact and time-based cleanupHybrid scenarios

How Retention Works:

  1. Kafka stores messages in segment files
  2. When segment age > retention.ms OR log size > retention.bytes
  3. Old segments are marked for deletion
  4. Cleaner thread removes eligible segments

🔍 Interviewer Notes: Testing operational configuration knowledge.

❌ Common Mistakes:

  • Thinking messages are deleted after consumption
  • Not sizing retention for recovery needs
  • Not understanding compact vs delete

Q8: What are Kafka consumer groups and how do they work?​

Technical Answer: Consumer groups enable parallel processing with automatic partition assignment.

Consumer Group Mechanics:

from kafka import KafkaConsumer

# Multiple consumers in same group
consumer = KafkaConsumer(
'orders',
bootstrap_servers=bootstrap_servers,
group_id='payment-processors', # Same group = load balanced
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM',
auto_offset_reset='earliest',
enable_auto_commit=True,
auto_commit_interval_ms=5000
)

for message in consumer:
process_payment(message.value)

Partition Assignment:

Topic: orders (6 partitions)

Consumer Group: payment-processors
┌─────────────────────────────────────────┐
│ Consumer 1: [P0, P1] │
│ Consumer 2: [P2, P3] │
│ Consumer 3: [P4, P5] │
└─────────────────────────────────────────┘

If Consumer 2 fails:
┌─────────────────────────────────────────┐
│ Consumer 1: [P0, P1, P2] │
│ Consumer 3: [P3, P4, P5] │
│ Consumer 2: [FAILED - rebalancing] │
└─────────────────────────────────────────┘

Key Behaviors:

  • Each partition assigned to exactly one consumer in group
  • Adding consumers triggers rebalancing
  • Consumer offset stored in __consumer_offsets topic
  • Rebalance strategies: Range (default), RoundRobin, Sticky, CooperativeSticky

🔍 Interviewer Notes: Testing distributed processing understanding.

❌ Common Mistakes:

  • Having more consumers than partitions (idle consumers)
  • Not handling rebalances properly
  • Not understanding offset commit implications

Q9: What is Kafka message ordering guarantee?​

Technical Answer: Kafka provides ordering guarantees at the partition level, not topic level.

Ordering Guarantees:

from kafka import KafkaProducer

producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM'
)

# Order guaranteed WITHIN partition
# Messages sent to same partition in order they were sent
producer.send('orders', key=b'user-123', value=b'order-1')
producer.send('orders', key=b'user-123', value=b'order-2')
producer.send('orders', key=b'user-123', value=b'order-3')

# If no key, round-robin partitioning = NO ordering across partition
producer.send('orders', value=b'order-a') # Might go to P0
producer.send('orders', value=b'order-b') # Might go to P1

Ordering Scenarios:

ScenarioOrdering Guarantee
Same producer, same partitionStrict ordering
Same producer, different partitionsNo ordering guarantee
Multiple producersNo ordering guarantee (wall-clock time)
Retries with max.in.flight.requests > 1Potential reordering on retry

For Strict Ordering:

  1. Use same partition key for related messages
  2. Set max.in.flight.requests.per.connection=1 (or use idempotent producer)
  3. Use single partition topic (limits throughput)

🔍 Interviewer Notes: Testing understanding of consistency vs throughput tradeoffs.

❌ Common Mistakes:

  • Assuming topic-level ordering
  • Not using partition keys for related events
  • Not considering retry reordering

Q10: How do you configure Kafka producer for reliability?​

Technical Answer: Producer configuration controls durability, availability, and performance tradeoffs.

Reliability Configurations:

from kafka import KafkaProducer

# High reliability configuration
reliable_producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM',

# Durability
acks='all', # Wait for all ISRs to confirm
retries=10, # Retry on transient failures
retry_backoff_ms=1000,

# Idempotence (exactly-once semantics)
enable_idempotence=True,

# Batching
batch_size=16384, # 16KB batches
linger_ms=5, # Wait up to 5ms to batch
compression_type='snappy',

# Buffering
buffer_memory=67108864, # 64MB buffer
max_block_ms=60000 # Block up to 60s when buffer full
)

# Send with callback for confirmation
def on_send_success(record_metadata):
print(f"Sent to {record_metadata.topic} "
f"partition {record_metadata.partition} "
f"offset {record_metadata.offset}")

def on_send_error(excp):
print(f"Send failed: {excp}")

producer.send('orders', b'order-data')\
.add_callback(on_send_success)\
.add_errback(on_send_error)

acks Setting Tradeoffs:

acksBehaviorDurabilityLatency
0Fire and forget, no confirmationLowLowest
1Wait for leader acknowledgmentMediumLow
allWait for all in-sync replicasHighHigher

🔍 Interviewer Notes: Testing production configuration knowledge.

❌ Common Mistakes:

  • Using acks=0 or acks=1 for critical data
  • Not enabling idempotence
  • Not handling send errors

Q11: How do you manage Kafka offsets?​

Technical Answer: Offsets track consumer position in a partition. Proper offset management ensures reliable message processing.

Offset Management Strategies:

from kafka import KafkaConsumer, TopicPartition

# Strategy 1: Auto-commit (simple, may lose messages on crash)
consumer = KafkaConsumer(
'orders',
bootstrap_servers=bootstrap_servers,
group_id='auto-commit-group',
enable_auto_commit=True,
auto_commit_interval_ms=5000 # Commit every 5 seconds
)

# Strategy 2: Manual commit (precise control)
consumer = KafkaConsumer(
'orders',
bootstrap_servers=bootstrap_servers,
group_id='manual-commit-group',
enable_auto_commit=False # Disable auto-commit
)

for message in consumer:
try:
process_message(message)
# Commit only after successful processing
consumer.commit_sync() # Blocks until committed
# Or use commit_async() for better throughput
except Exception as e:
# Don't commit - message will be reprocessed
print(f"Processing failed: {e}")
continue

# Strategy 3: Store offsets externally (exactly-once with DB)
# Commit to DB along with processed data

Offset Reset Behavior:

auto.offset.resetBehavior
earliestStart from beginning
latestStart from end (new messages only)
noneThrow exception if no offset

🔍 Interviewer Notes: Testing reliability and consistency understanding.

❌ Common Mistakes:

  • Auto-committing before processing (lose messages on crash)
  • Not handling commit failures
  • Wrong offset reset for use case

Q12: What is Kafka replication and how does it work?​

Technical Answer: Replication ensures durability by copying partition data to multiple brokers.

Replication Concepts:

from kafka.admin import KafkaAdminClient, NewTopic

# Create topic with replication
admin_client = KafkaAdminClient(
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM'
)

new_topic = NewTopic(
name='critical-orders',
num_partitions=6,
replication_factor=3, # 3 copies of each partition
topic_configs={
'min.insync.replicas': '2' # Require 2 replicas for acks=all
}
)

admin_client.create_topics([new_topic])

Replication Factor Tradeoffs:

RFDurabilityWrite LatencyStorage CostAvailability
1LowLowest1xLow
2MediumMedium2xMedium
3HighHigher3xHigh

ISR (In-Sync Replicas):

  • Leader + followers that are caught up
  • min.insync.replicas determines minimum ISR size for writes
  • If ISR drops below minimum, producer with acks=all will fail

🔍 Interviewer Notes: Testing durability and availability tradeoffs.

❌ Common Mistakes:

  • RF=1 for critical data
  • Not setting min.insync.replicas
  • Not monitoring ISR size

Q13: How do you handle poison pill messages in Kafka?​

Technical Answer: Poison pill messages are malformed records that crash consumers. Need strategies to handle them.

Handling Strategies:

from kafka import KafkaConsumer
import json
import logging

def safe_deserialize(value):
"""Safe deserialization with error handling"""
try:
if value is None:
return None
return json.loads(value.decode('utf-8'))
except (json.JSONDecodeError, UnicodeDecodeError) as e:
logging.error(f"Deserialization failed: {e}, value: {value}")
return {'__error__': True, '__raw__': value}

consumer = KafkaConsumer(
'orders',
bootstrap_servers=bootstrap_servers,
group_id='robust-consumer',
value_deserializer=safe_deserialize
)

dlq_producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM'
)

for message in consumer:
data = message.value

# Check for deserialization errors
if data and data.get('__error__'):
# Send to DLQ without retrying
dlq_producer.send('orders-dlq', {
'original_message': data['__raw__'],
'error': 'deserialization_failed',
'timestamp': time.time()
})
consumer.commit_sync()
continue

try:
process_order(data)
consumer.commit_sync()
except NonRetryableError as e:
# Business logic error - send to DLQ
dlq_producer.send('orders-dlq', {
'original_message': data,
'error': str(e),
'error_type': 'business_logic'
})
consumer.commit_sync()
except RetryableError as e:
# Transient error - retry without committing
logging.warning(f"Retryable error: {e}")
continue

🔍 Interviewer Notes: Testing error handling and operational resilience.

❌ Common Mistakes:

  • No poison pill handling (infinite crash loop)
  • Not using DLQs
  • Discarding messages instead of analyzing

Q14: What is Kafka log compaction?​

Technical Answer: Log compaction retains the latest value for each key, removing older versions. Useful for changelog topics.

Compaction Configuration:

from kafka.admin import KafkaAdminClient, ConfigResource, ConfigResourceType

admin_client = KafkaAdminClient(
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM'
)

# Configure compacted topic
config_resources = [ConfigResource(
ConfigResourceType.TOPIC,
'user-profiles',
configs={
'cleanup.policy': 'compact',
'min.cleanable.dirty.ratio': '0.5', # Compact when 50% dirty
'delete.retention.ms': '86400000', # Keep tombstones 1 day
'min.compaction.lag.ms': '0', # Compact immediately
'max.compaction.lag.ms': '604800000' # Max 7 days before compact
}
)]

admin_client.alter_configs(config_resources)

Compaction Use Cases:

  • State store changelog (KTable)
  • Configuration/settings updates
  • User profile changes
  • Any "latest state" scenario

How It Works:

  1. Kafka marks segments as clean or dirty
  2. When dirty ratio > threshold, compaction triggers
  3. Cleaner thread builds map of latest offset per key
  4. Copies latest values to new segment, deletes old

🔍 Interviewer Notes: Testing understanding of Kafka storage mechanisms.

❌ Common Mistakes:

  • Using compaction for event streams
  • Not setting proper retention for tombstones
  • Not understanding throughput impact

Q15: How do you size Kafka partitions?​

Technical Answer: Proper partition sizing balances throughput, parallelism, and overhead.

Partition Sizing Factors:

# Factors to consider:
# 1. Target throughput
# 2. Producer throughput per partition: ~10MB/s
# 3. Consumer throughput per partition: ~10MB/s
# 4. Desired parallelism

def calculate_partitions(target_throughput_mbps,
producer_rate=10,
consumer_rate=10,
consumer_count=10):
"""
Calculate optimal partition count

target_throughput_mbps: Desired MB/s for topic
producer_rate: MB/s per partition for producers
consumer_rate: MB/s per partition for consumers
consumer_count: Number of consumer instances
"""
# Producer constraint
partitions_by_producer = target_throughput_mbps / producer_rate

# Consumer constraint
partitions_by_consumer = target_throughput_mbps / consumer_rate

# Consumer parallelism constraint
# Need at least as many partitions as consumers
partitions_by_parallelism = consumer_count

# Choose max of all constraints
optimal_partitions = max(
partitions_by_producer,
partitions_by_consumer,
partitions_by_parallelism
)

return int(optimal_partitions)

# Example: 100 MB/s target, 10 consumers
partitions = calculate_partitions(100, consumer_count=10)
print(f"Optimal partitions: {partitions}") # Result: 10

Partition Limits:

  • MSK: Up to 2000 partitions per broker
  • More partitions = more overhead
  • Each partition = 20MB memory on broker
  • Rebalancing time increases with partitions

🔍 Interviewer Notes: Testing capacity planning and throughput optimization.

❌ Common Mistakes:

  • Too few partitions (throughput bottleneck)
  • Too many partitions (rebalancing overhead)
  • Not considering consumer count

Q16: What are Kafka transactions?​

Technical Answer: Kafka transactions enable atomic multi-partition writes and exactly-once processing.

Transaction Basics:

from kafka import KafkaProducer

# Create transactional producer
producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM',
transactional_id='order-producer-1', # Unique per producer
enable_idempotence=True # Required for transactions
)

# Initialize transactions
producer.init_transactions()

def process_payment(order_id):
try:
# Begin transaction
producer.begin_transaction()

# Send to multiple topics atomically
producer.send('payments', key=order_id, value={'status': 'processed'})
producer.send('notifications', key=order_id, value={'message': 'Payment received'})
producer.send('audit-log', key=order_id, value={'action': 'payment_processed'})

# Commit all or nothing
producer.commit_transaction()

except Exception as e:
producer.abort_transaction()
raise

Transaction Guarantees:

  • Atomic multi-partition writes
  • All commits or none
  • Consumers see committed messages only (isolation.level=read_committed)

🔍 Interviewer Notes: Testing exactly-once semantics understanding.

❌ Common Mistakes:

  • Not using idempotent producers with transactions
  • Duplicate transactional_id (causes fencing)
  • Not handling transaction timeout

Q17: How do you use Kafka Streams for stream processing?​

Technical Answer: Kafka Streams is a client library for building stream processing applications.

Basic Streams Application:

// Java example - Kafka Streams is Java/Scala library
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "word-count-app");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());

StreamsBuilder builder = new StreamsBuilder();

KStream<String, String> source = builder.stream("input-topic");

// Word count example
KTable<String, Long> wordCounts = source
.flatMapValues(value -> Arrays.asList(value.toLowerCase().split("\\W+")))
.groupBy((key, value) -> value)
.count(Materialized.as("counts-store"));

wordCounts.toStream().to("output-topic", Produced.with(Serdes.String(), Serdes.Long()));

KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();

Kafka Streams Features:

  • Stateful and stateless processing
  • Event-time processing with windowing
  • Exactly-once semantics
  • No separate processing cluster needed

🔍 Interviewer Notes: Testing stream processing knowledge.

❌ Common Mistakes:

  • Not understanding KStream vs KTable
  • Wrong serialization configuration
  • Not handling rebalancing properly

Q18: How do you implement Kafka security?​

Technical Answer: Kafka security includes encryption, authentication, and authorization.

Security Layers:

# TLS/SSL Encryption
producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL', # or 'SSL' for mTLS
ssl_cafile='msk-certificate.pem',
ssl_check_hostname=True
)

# SASL/SCRAM Authentication
producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL',
sasl_mechanism='SCRAM-SHA-512',
sasl_plain_username='alice',
sasl_plain_password='secret-password'
)

# AWS MSK IAM Authentication
producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM',
sasl_jaas_config='software.amazon.msk.auth.iam.IAMLoginModule required;',
sasl_client_callback_handler_class='software.amazon.msk.auth.iam.IAMClientCallbackHandler'
)

ACL Authorization:

# Grant Alice permission to produce to topic
kafka-acls.sh --bootstrap-server $BS \
--add --allow-principal User:alice \
--operation Write --topic orders

# Grant consumer group read permission
kafka-acls.sh --bootstrap-server $BS \
--add --allow-principal User:bob \
--operation Read --topic orders \
--group order-processors

🔍 Interviewer Notes: Testing security best practices.

❌ Common Mistakes:

  • Not enabling TLS in transit
  • Using PLAINTEXT in production
  • Not setting up proper ACLs

Q19: What is Kafka Connect?​

Technical Answer: Kafka Connect is a framework for streaming data between Kafka and external systems.

Connect Architecture:

┌─────────────────────────────────────────────────────────────┐
│ Kafka Connect │
│ ┌─────────────────────┐ ┌─────────────────────┐ │
│ │ Source Connector │ │ Sink Connector │ │
│ │ (Database → Kafka)│ │ (Kafka → S3) │ │
│ └──────────┬──────────┘ └──────────┬──────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌─────────────────────┐ ┌─────────────────────┐ │
│ │ Debezium CDC │ │ S3 Sink Connector │ │
│ │ JDBC Source │ │ Elasticsearch │ │
│ └─────────────────────┘ └─────────────────────┘ │
│ │ │
│ Kafka Topic │
└─────────────────────────────────────────────────────────────┘

Common Connectors:

  • Source: JDBC, Debezium (CDC), FileStream, S3
  • Sink: S3, Elasticsearch, JDBC, BigQuery, Snowflake

🔍 Interviewer Notes: Testing data integration patterns.

❌ Common Mistakes:

  • Not using Connect for simple integrations
  • Wrong connector for use case
  • Not handling schema evolution

Q20: How do you handle Kafka schema evolution?​

Technical Answer: Schema evolution allows data format changes without breaking consumers.

Schema Compatibility Modes:

# Using Schema Registry
from confluent_kafka import Producer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer

schema_registry_conf = {'url': 'http://schema-registry:8081'}
schema_registry_client = SchemaRegistryClient(schema_registry_conf)

# Define Avro schema with backward compatibility
order_schema = """
{
"type": "record",
"name": "Order",
"fields": [
{"name": "order_id", "type": "string"},
{"name": "customer_id", "type": "string"},
{"name": "amount", "type": "double"},
{"name": "currency", "type": "string", "default": "USD"}, # Added with default
{"name": "timestamp", "type": "long"}
]
}
"""

avro_serializer = AvroSerializer(
schema_registry_client,
order_schema,
lambda order, ctx: {
'order_id': order['id'],
'customer_id': order['customer'],
'amount': order['amount'],
'timestamp': order['timestamp']
}
)

producer = Producer({
'bootstrap.servers': bootstrap_servers,
'key.serializer': None,
'value.serializer': avro_serializer
})

Compatibility Types:

TypeDescriptionUse Case
BACKWARDNew schema can read old dataDefault, safe
FORWARDOld schema can read new dataLess common
FULLBoth backward and forwardIdeal but restrictive
NONENo compatibility checksNot recommended

🔍 Interviewer Notes: Testing data governance and schema management.

❌ Common Mistakes:

  • Not using Schema Registry
  • Breaking compatibility without planning
  • Not versioning schemas properly

Q21: What is a Kafka offset commit?​

Technical Answer: Offset commits record the last successfully processed message position. Critical for exactly-once or at-least-once semantics.

Commit Strategies:

from kafka import KafkaConsumer, TopicPartition

# At-most-once (auto-commit, risk of data loss)
consumer = KafkaConsumer('orders',
enable_auto_commit=True,
auto_commit_interval_ms=5000)

# At-least-once (manual commit after processing)
consumer = KafkaConsumer('orders',
enable_auto_commit=False)

for msg in consumer:
process(msg)
consumer.commit_sync() # Commit after success

# Exactly-once (transactional with store-and-forward)
# Store offset with processed result in external DB

🔍 Interviewer Notes: Testing message delivery guarantees understanding.

❌ Common Mistakes:

  • Auto-committing before processing
  • Not handling commit failures
  • Wrong delivery semantics for use case

Q22: How do Kafka consumer rebalances work?​

Technical Answer: Rebalances redistribute partitions when consumers join or leave the group.

Rebalance Triggers:

  • Consumer joins group
  • Consumer leaves (graceful or crash)
  • Subscription changes
  • Coordinator detects heartbeat timeout

Strategies:

  • Eager (default): Stop all processing, reassign, resume
  • Cooperative: Revoke subset, process remaining, incrementally reassign
consumer = KafkaConsumer('orders',
partition_assignment_strategy='org.apache.kafka.clients.consumer.CooperativeStickyAssignor')

🔍 Interviewer Notes: Testing understanding of distributed coordination.

❌ Common Mistakes:

  • Not handling rebalance in processing logic
  • Long processing causing heartbeat timeouts
  • Not using cooperative strategy (causes stop-world)

Q23: What is the Kafka log structure?​

Technical Answer: Kafka stores messages in append-only log files called segments.

Storage Structure:

/kafka-logs/topic-orders-0/
├── 00000000000000000000.log # Segment 0: offset 0-999999
├── 00000000000000000000.index # Offset to position mapping
├── 00000000000000000000.timeindex # Timestamp to offset
├── 00000000000010000000.log # Segment 1: offset 1000000+
├── 00000000000010000000.index
└── 00000000000010000000.timeindex

Segment Rolling:

  • By size (1GB default)
  • By time (7 days default)
  • Manual trigger

🔍 Interviewer Notes: Testing storage architecture knowledge.

❌ Common Mistakes:

  • Not understanding segment impact on retention
  • Wrong segment sizing for workload

Q24: How do you configure Kafka consumer fetch behavior?​

Technical Answer: Fetch configuration affects throughput and latency tradeoffs.

consumer = KafkaConsumer('orders',
# Fetch min/max bytes
fetch_min_bytes=1, # Fetch at least 1 byte
fetch_max_bytes=52428800, # Max 50MB per fetch

# Partition-level limits
max_partition_fetch_bytes=1048576, # Max 1MB per partition

# Wait time
fetch_max_wait_ms=500, # Wait up to 500ms for fetch_min_bytes

# Poll behavior
max_poll_records=500, # Max records per poll()
max_poll_interval_ms=300000 # Max time between polls
)

🔍 Interviewer Notes: Testing performance tuning knowledge.

❌ Common Mistakes:

  • max_poll_interval_ms too short for long processing
  • fetch_max_bytes too small (frequent fetches)
  • Not considering partition limits

Q25: What are Kafka record keys and why use them?​

Technical Answer: Keys determine partition routing and enable log compaction.

Key Functions:

  1. Partition Routing: partition = hash(key) % num_partitions
  2. Ordering: Same key → same partition → ordering
  3. Compaction: Latest record per key retained
# With key - deterministic routing, ordering
producer.send('orders', key='user-123', value=order)

# Without key - round-robin routing
producer.send('orders', value=order)

# Null key for compacted topics (deletion tombstone)
producer.send('user-profiles', key='user-123', value=None)

🔍 Interviewer Notes: Testing data modeling and routing understanding.

❌ Common Mistakes:

  • Not using keys when ordering needed
  • Using high-cardinality keys (poor distribution)
  • Not understanding null key behavior

Q26: How do you configure Kafka for high availability?​

Technical Answer: High availability requires proper replication, ISR settings, and broker distribution.

from kafka.admin import KafkaAdminClient, NewTopic

admin_client = KafkaAdminClient(
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM'
)

# Create topic with HA settings
topic = NewTopic(
name='critical-events',
num_partitions=6,
replication_factor=3, # 3 copies
topic_configs={
'min.insync.replicas': '2', # Write to 2 brokers minimum
'unclean.leader.election.enable': 'false' # Prevent data loss
}
)

admin_client.create_topics([topic])

HA Checklist:

  • Replication factor ≥ 3
  • Min in-sync replicas ≥ 2
  • Brokers across multiple AZs
  • Unclean leader election disabled

🔍 Interviewer Notes: Testing production reliability knowledge.

❌ Common Mistakes:

  • RF=2 (can't tolerate 2 failures)
  • min.insync.replicas=1 (ack=all still weak)
  • Enabling unclean leader election

Q27: What is Kafka's default partitioning strategy?​

Technical Answer: Default partitioner uses murmur2 hash of key modulo partition count.

from kafka import KafkaProducer

producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
key_serializer=lambda k: k.encode('utf-8'),
value_serializer=lambda v: json.dumps(v).encode('utf-8')
)

# With key - hash partitioning
producer.send('orders', key='user-123', value={'order': 1})
# Same key → same partition

# Without key - round-robin (sticky batching in 2.4+)
producer.send('orders', value={'order': 2})
producer.send('orders', value={'order': 3})

Partitioner Types:

PartitionerBehaviorUse Case
DefaultHash(key) % partitionsOrdering needed
RoundRobinEven distributionNo ordering needed
StickyBatches by partitionPerformance
CustomUser-definedSpecial routing

🔍 Interviewer Notes: Testing data distribution understanding.

❌ Common Mistakes:

  • Hot partitions (poor key selection)
  • Assuming round-robin with keys
  • Not testing partition distribution

Q28: How do you monitor Kafka consumer lag?​

Technical Answer: Consumer lag measures how far behind consumers are from the latest message.

from kafka.admin import KafkaAdminClient

admin_client = KafkaAdminClient(
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM'
)

# Get consumer group lag
from kafka import KafkaConsumer

consumer = KafkaConsumer(
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM'
)

# List consumer groups and their lag
for group in consumer.list_consumer_groups():
group_id = group[0]
lag_info = consumer.list_consumer_group_offsets(group_id)

for tp, offset_metadata in lag_info.items():
committed = offset_metadata.offset
# Get end offset for partition
end_offsets = consumer.end_offsets([tp])
lag = end_offsets[tp] - committed

print(f"Group: {group_id}, Topic: {tp.topic}, "
f"Partition: {tp.partition}, Lag: {lag}")

consumer.close()

Lag Alerting (CloudWatch):

cloudwatch.put_metric_alarm(
AlarmName='kafka-consumer-lag',
MetricName='SumOffsetLag',
Namespace='AWS/Kafka',
Statistic='Average',
Period=300,
EvaluationPeriods=2,
Threshold=10000,
ComparisonOperator='GreaterThanThreshold',
Dimensions=[
{'Name': 'Cluster Name', 'Value': 'my-cluster'},
{'Name': 'Consumer Group', 'Value': 'order-processors'}
]
)

🔍 Interviewer Notes: Testing operational monitoring.

❌ Common Mistakes:

  • Not monitoring lag proactively
  • Ignoring growing lag trends
  • Not scaling consumers when lag increases

Q29: How do you handle Kafka broker failures?​

Technical Answer: Kafka handles broker failures automatically through leader election and replication.

Failure Scenarios:

Broker Failure - Automatic Recovery:
1. Controller detects broker failure (missing heartbeats)
2. Identifies affected partitions (leaders on failed broker)
3. Elects new leaders from ISRs
4. Updates metadata and notifies clients
5. Clients reconnect to new leaders

┌───────────────────────────────────────┐
│ Before Failure: │
│ P0: Leader=B1, Replicas=[B1,B2,B3] │
│ P1: Leader=B2, Replicas=[B2,B3,B1] │
│ P2: Leader=B3, Replicas=[B3,B1,B2] │
├───────────────────────────────────────┤
│ After B1 Failure: │
│ P0: Leader=B2 (elected from ISR) │
│ P1: Leader=B2 (unchanged) │
│ P2: Leader=B3 (unchanged) │
└───────────────────────────────────────┘

Handling Tips:

  • Don't panic - Kafka is designed for failures
  • Monitor under-replicated partitions
  • Ensure min.insync.replicas < replication.factor
  • Have monitoring alerts for broker downtime

🔍 Interviewer Notes: Testing distributed systems understanding.

❌ Common Mistakes:

  • Manually intervening too quickly
  • Not waiting for automatic recovery
  • Setting min.insync.replicas = replication.factor (blocks on 1 failure)

Q30: How do you optimize Kafka producer throughput?​

Technical Answer: Optimize batching, compression, and asynchronous sends.

from kafka import KafkaProducer

producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM',

# Batching - key for throughput
batch_size=65536, # 64KB batches (default 16KB)
linger_ms=10, # Wait up to 10ms to batch

# Compression
compression_type='snappy', # snappy, lz4, gzip, zstd

# Buffering
buffer_memory=67108864, # 64MB (default 32MB)
max_block_ms=60000, # Block up to 60s if buffer full

# Async - don't wait for response
acks='1', # Or 'all' for durability
retries=3,
max_in_flight_requests_per_connection=5
)

# Send asynchronously with callback
def on_success(record_metadata):
print(f"Sent to partition {record_metadata.partition}")

def on_error(exception):
print(f"Error: {exception}")

for i in range(10000):
producer.send('orders', key=f'user-{i%100}', value={'order': i})\
.add_callback(on_success)\
.add_errback(on_error)

# Flush remaining messages
producer.flush()

Throughput Checklist:

  • Increase batch_size (32KB-128KB)
  • Set linger_ms (5-100ms)
  • Enable compression (snappy for speed, zstd for ratio)
  • Increase buffer_memory for burst handling
  • Use async sends with callbacks

🔍 Interviewer Notes: Testing performance optimization knowledge.

❌ Common Mistakes:

  • Synchronous sends (waiting for each ack)
  • No compression (high network usage)
  • batch_size too small
  • Not handling backpressure (buffer full)

Q31: What are Kafka topic configuration best practices?​

Technical Answer: Topic configuration affects performance, retention, and reliability.

from kafka.admin import KafkaAdminClient, NewTopic

admin_client = KafkaAdminClient(
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM'
)

# Create topic with optimized settings
topic = NewTopic(
name='events',
num_partitions=12,
replication_factor=3,
topic_configs={
# Retention (time or size based)
'retention.ms': '604800000', # 7 days
'retention.bytes': '-1', # No size limit

# Segment settings
'segment.ms': '604800000', # Roll segment every 7 days
'segment.bytes': '1073741824', # Or 1GB

# Cleanup
'cleanup.policy': 'delete', # or 'compact', 'compact,delete'
'min.cleanable.dirty.ratio': '0.5',

# Compression
'compression.type': 'producer', # Use producer's setting

# Message size
'max.message.bytes': '1048576', # 1MB max message

# Replication
'min.insync.replicas': '2',
'unclean.leader.election.enable': 'false'
}
)

admin_client.create_topics([topic])

Configuration by Use Case:

Use Caseretentioncleanupcompression
Event streaming7 daysdeletesnappy
Audit logs1 yeardeletezstd
State changes30 dayscompactlz4
High throughput3 daysdeletesnappy

🔍 Interviewer Notes: Testing operational configuration.

❌ Common Mistakes:

  • Default retention too short for recovery needs
  • Wrong cleanup policy for use case
  • Not setting message size limits

Q42: How do you enable encryption at rest for Amazon MSK clusters?​

Technical Answer: MSK encrypts data at rest using AWS KMS keys to protect broker EBS volumes and stored messages. You can choose AWS managed keys (default) or customer-managed KMS keys (CMKs) for compliance and access control.

Code/Config Example:

import boto3

kafka = boto3.client('kafka')

response = kafka.create_cluster(
ClusterName='encrypted-msk-cluster',
KafkaVersion='3.5.1',
NumberOfBrokerNodes=3,
BrokerNodeGroupInfo={
'InstanceType': 'kafka.m5.large',
'ClientSubnets': ['subnet-123', 'subnet-456', 'subnet-789'],
'SecurityGroups': ['sg-123'],
'StorageInfo': {'EbsStorageInfo': {'VolumeSize': 1000}}
},
EncryptionInfo={
'EncryptionAtRest': {
'DataVolumeKMSKeyId': 'arn:aws:kms:us-east-1:123456789:key/1234abcd-12ab-34cd-56ef-1234567890ab'
},
'EncryptionInTransit': {'InCluster': True, 'ClientBroker': 'TLS'}
}
)

🔍 Interviewer Notes: Tests understanding of MSK data protection and KMS integration.

❌ Common Mistakes:

  • Not enabling encryption at rest for production workloads
  • Using default AWS managed keys when CMKs are required for compliance
  • Forgetting to grant MSK service role access to KMS keys

Q43: What TLS settings are required for MSK client-broker encryption?​

Technical Answer: MSK supports TLS encryption for data in transit between clients and brokers, with options for TLS, PLAINTEXT, or both. Enforce TLS only for production to prevent unencrypted data transmission.

Code/Config Example:

from kafka import KafkaProducer
from aws_msk_iam_sasl_signer import MSKAuthTokenProvider

def get_auth_token():
return MSKAuthTokenProvider.generate_auth_token('us-east-1')

producer = KafkaProducer(
bootstrap_servers=['b-1.my-cluster.kafka.us-east-1.amazonaws.com:9098'],
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM',
sasl_aws_msk_iam_auth_token_provider=get_auth_token,
ssl_cafile='AmazonRootCA1.pem'
)

🔍 Interviewer Notes: Tests knowledge of MSK in-transit encryption requirements.

❌ Common Mistakes:

  • Using PLAINTEXT security protocol in production
  • Missing CA certificate for TLS verification
  • Not matching client security protocol to MSK cluster settings

Q44: How do you configure MSK inter-broker encryption?​

Technical Answer: Inter-broker encryption secures data transmitted between MSK brokers using TLS. This is enabled by default for MSK clusters and cannot be disabled for provisioned clusters.

Code/Config Example:

import boto3

kafka = boto3.client('kafka')

response = kafka.describe_cluster(
ClusterArn='arn:aws:kafka:us-east-1:123456789:cluster/my-cluster/abcd1234'
)

inter_broker_encryption = response['ClusterInfo']['EncryptionInfo']['EncryptionInTransit']['InCluster']
print(f"Inter-broker TLS enabled: {inter_broker_encryption}") # True by default

🔍 Interviewer Notes: Tests understanding of MSK default security features.

❌ Common Mistakes:

  • Assuming inter-broker encryption needs manual configuration
  • Not verifying encryption settings for compliance
  • Confusing client-broker and inter-broker encryption

Q45: What is AWS KMS integration with MSK encryption?​

Technical Answer: MSK uses KMS to manage encryption keys for data at rest and in-transit certificate management. Customer-managed KMS keys let you control key rotation, access policies, and audit key usage.

Code/Config Example:

import boto3
import json

kms = boto3.client('kms')

# Create CMK for MSK encryption
kms.create_key(
Description='MSK cluster encryption key',
KeyUsage='ENCRYPT_DECRYPT',
Origin='AWS_KMS',
Policy=json.dumps({
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"Service": "kafka.amazonaws.com"},
"Action": ["kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*"],
"Resource": "*"
}]
})
)

🔍 Interviewer Notes: Tests KMS and MSK integration knowledge.

❌ Common Mistakes:

  • Not granting MSK service principal access to KMS keys
  • Using same KMS key for multiple clusters without rotation
  • Not enabling automatic key rotation

Q46: How do you enforce TLS 1.2 or higher for MSK clients?​

Technical Answer: MSK enforces TLS 1.2 or higher for client connections by default when using TLS security protocol. You can further restrict TLS versions via client-side configuration or security group rules.

Code/Config Example:

import ssl
from kafka import KafkaProducer

producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL',
ssl_cafile='AmazonRootCA1.pem',
ssl_version=ssl.PROTOCOL_TLSv1_2,
ssl_check_hostname=True
)

🔍 Interviewer Notes: Tests TLS version enforcement and client configuration.

❌ Common Mistakes:

  • Not specifying TLS version on clients
  • Allowing older TLS versions via security groups
  • Disabling hostname verification

Q47: What data protection features does MSK offer beyond encryption?​

Technical Answer: MSK provides IAM authentication for access control, VPC isolation for network security, and ACLs for fine-grained topic permissions. These features combine to protect data from unauthorized access.

Code/Config Example:

# IAM policy for restricted MSK access
msk_iam_policy = {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["kafka-cluster:Connect"],
"Resource": "arn:aws:kafka:us-east-1:123456789:cluster/my-cluster/*"
}, {
"Effect": "Deny",
"Action": ["kafka-cluster:Read", "kafka-cluster:Write"],
"Resource": "arn:aws:kafka:us-east-1:123456789:topic/my-cluster/*/sensitive-data"
}]
}

🔍 Interviewer Notes: Tests holistic data protection understanding.

❌ Common Mistakes:

  • Relying only on encryption without access controls
  • Not using VPC endpoints for private cluster access
  • Overly permissive IAM policies

Q48: How do you encrypt MSK broker logs and metrics?​

Technical Answer: MSK broker logs (to CloudWatch Logs) and metrics (to CloudWatch) are encrypted in transit by default. For at-rest encryption of logs, enable CloudWatch Logs encryption with KMS.

Code/Config Example:

import boto3

logs = boto3.client('logs')

# Enable KMS encryption for CloudWatch Log group
logs.associate_kms_key(
logGroupName='/aws/msk/my-cluster',
kmsKeyId='arn:aws:kms:us-east-1:123456789:key/1234abcd-12ab-34cd-56ef-1234567890ab'
)

🔍 Interviewer Notes: Tests log and metric security knowledge.

❌ Common Mistakes:

  • Assuming broker logs are encrypted at rest by default
  • Not encrypting CloudWatch Logs for compliance
  • Missing KMS permissions for log encryption

Q49: What is mutual TLS (mTLS) authentication for MSK?​

Technical Answer: mTLS requires clients to present a valid certificate to brokers for authentication, in addition to broker certificate verification. MSK supports mTLS via SASL/SCRAM or custom certificate authorities.

Code/Config Example:

from kafka import KafkaProducer

producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
security_protocol='SSL',
ssl_cafile='broker-ca.pem',
ssl_certfile='client-cert.pem',
ssl_keyfile='client-key.pem'
)

🔍 Interviewer Notes: Tests advanced authentication mechanisms.

❌ Common Mistakes:

  • Confusing mTLS with SASL/SSL
  • Not managing client certificate rotation
  • Using self-signed certificates without proper CA setup

Q50: How do you rotate encryption keys for MSK clusters?​

Technical Answer: For KMS keys, enable automatic rotation (annual) or manually rotate keys by updating the cluster's EncryptionAtRest configuration. MSK manages broker TLS certificates automatically; client certificates require manual rotation.

Code/Config Example:

import boto3

kms = boto3.client('kms')
kafka = boto3.client('kafka')

# Enable automatic KMS key rotation
kms.enable_key_rotation(KeyId='arn:aws:kms:us-east-1:123456789:key/1234abcd-12ab-34cd-56ef-1234567890ab')

# Update MSK cluster to new KMS key
kafka.update_cluster_configuration(
ClusterArn='arn:aws:kafka:us-east-1:123456789:cluster/my-cluster/abcd1234',
ConfigurationInfo={
'EncryptionInfo': {
'EncryptionAtRest': {'DataVolumeKMSKeyId': 'new-kms-key-arn'}
}
}
)

🔍 Interviewer Notes: Tests key rotation and compliance workflows.

❌ Common Mistakes:

  • Not rotating KMS keys regularly
  • Forgetting to update IAM policies for new keys
  • Assuming MSK rotates client certificates automatically

Q51: How do you monitor MSK clusters using CloudWatch?​

Technical Answer: MSK automatically publishes cluster, broker, and topic metrics to Amazon CloudWatch. You can visualize metrics in the CloudWatch console or query them programmatically via the Boto3 client.

Code/Config Example:

import boto3
from datetime import datetime, timedelta

cloudwatch = boto3.client('cloudwatch')

response = cloudwatch.get_metric_statistics(
Namespace='AWS/Kafka',
MetricName='BytesInPerSec',
Dimensions=[
{'Name': 'Cluster Name', 'Value': 'my-cluster'},
{'Name': 'Broker ID', 'Value': '1'}
],
StartTime=datetime.utcnow() - timedelta(hours=1),
EndTime=datetime.utcnow(),
Period=300,
Statistics=['Average', 'Maximum']
)

🔍 Interviewer Notes: Tests CloudWatch integration knowledge.

❌ Common Mistakes:

  • Missing broker ID dimension in metric queries
  • Not setting up baseline alerts
  • Ignoring topic-level metrics

Q52: What key CloudWatch metrics should you track for MSK?​

Technical Answer: Critical metrics include BytesInPerSec, BytesOutPerSec, UnderReplicatedPartitions, ConsumerLag, CpuUtilization, and MemoryUsed. These help identify performance bottlenecks and reliability risks.

Code/Config Example:

key_metrics = [
'BytesInPerSec', 'BytesOutPerSec', 'MessagesInPerSec',
'UnderReplicatedPartitions', 'CpuUtilization', 'MemoryUsed',
'ConsumerLag', 'FailedProduceRequests', 'FailedFetchRequests'
]

🔍 Interviewer Notes: Tests operational monitoring best practices.

❌ Common Mistakes:

  • Tracking too many metrics (high CloudWatch costs)
  • Ignoring under-replicated partition alerts
  • Not monitoring consumer lag proactively

Q53: How do you set up CloudWatch alarms for MSK consumer lag?​

Technical Answer: Create CloudWatch alarms using the SumOffsetLag metric filtered by cluster name and consumer group. Set thresholds based on your SLA (e.g., 10k lag for 5 minutes triggers alert).

Code/Config Example:

cloudwatch.put_metric_alarm(
AlarmName='msk-consumer-lag-high',
MetricName='SumOffsetLag',
Namespace='AWS/Kafka',
Dimensions=[
{'Name': 'Cluster Name', 'Value': 'my-cluster'},
{'Name': 'Consumer Group', 'Value': 'order-processors'}
],
Statistic='Average',
Period=300,
EvaluationPeriods=2,
Threshold=10000,
ComparisonOperator='GreaterThanThreshold',
AlarmActions=['arn:aws:sns:us-east-1:123456789:msk-alerts']
)

🔍 Interviewer Notes: Tests alerting and SLA management.

❌ Common Mistakes:

  • Setting thresholds too high/low
  • Not filtering by consumer group
  • No alarm actions configured

Q54: How do you monitor MSK broker CPU and memory usage?​

Technical Answer: Use the CpuUtilization and MemoryUsed metrics from the AWS/Kafka namespace. Broker IDs are required dimensions to track per-broker resource usage.

Code/Config Example:

# Get CPU utilization for all brokers
for broker_id in ['1', '2', '3']:
metrics = cloudwatch.get_metric_statistics(
Namespace='AWS/Kafka',
MetricName='CpuUtilization',
Dimensions=[
{'Name': 'Cluster Name', 'Value': 'my-cluster'},
{'Name': 'Broker ID', 'Value': broker_id}
],
StartTime=datetime.utcnow() - timedelta(hours=1),
EndTime=datetime.utcnow(),
Period=300,
Statistics=['Average', 'Maximum']
)
print(f"Broker {broker_id} CPU: {metrics['Datapoints']}")

🔍 Interviewer Notes: Tests resource monitoring skills.

❌ Common Mistakes:

  • Not tracking per-broker metrics
  • Ignoring memory pressure warnings
  • Setting CPU thresholds too high

Q55: What is the UnderReplicatedPartitions metric?​

Technical Answer: UnderReplicatedPartitions counts partitions where the number of in-sync replicas is less than the total replica count. This indicates broker failures or network issues.

Code/Config Example:

# Alarm for under-replicated partitions
cloudwatch.put_metric_alarm(
AlarmName='msk-under-replicated-partitions',
MetricName='UnderReplicatedPartitions',
Namespace='AWS/Kafka',
Dimensions=[{'Name': 'Cluster Name', 'Value': 'my-cluster'}],
Statistic='Maximum',
Period=60,
EvaluationPeriods=3,
Threshold=1,
ComparisonOperator='GreaterThanThreshold'
)

🔍 Interviewer Notes: Tests reliability monitoring knowledge.

❌ Common Mistakes:

  • Ignoring transient under-replication
  • Not investigating root cause
  • Setting threshold too high

Q56: How do you monitor MSK topic-level metrics?​

Technical Answer: Add the Topic Name dimension to CloudWatch metric queries to track per-topic throughput, lag, and error rates. Topic-level metrics help isolate issues to specific workloads.

Code/Config Example:

response = cloudwatch.get_metric_statistics(
Namespace='AWS/Kafka',
MetricName='BytesInPerSec',
Dimensions=[
{'Name': 'Cluster Name', 'Value': 'my-cluster'},
{'Name': 'Topic Name', 'Value': 'orders'}
],
StartTime=datetime.utcnow() - timedelta(hours=1),
EndTime=datetime.utcnow(),
Period=300,
Statistics=['Average']
)

🔍 Interviewer Notes: Tests granular monitoring skills.

❌ Common Mistakes:

  • Forgetting to add Topic Name dimension
  • Not monitoring high-traffic topics separately
  • Missing error metrics per topic

Q57: How do you track MSK broker leader count?​

Technical Answer: The LeaderCount metric shows the number of partition leaders hosted on a broker. Imbalanced leader counts indicate uneven partition distribution.

Code/Config Example:

# Check leader distribution
for broker_id in ['1', '2', '3']:
metrics = cloudwatch.get_metric_statistics(
Namespace='AWS/Kafka',
MetricName='LeaderCount',
Dimensions=[
{'Name': 'Cluster Name', 'Value': 'my-cluster'},
{'Name': 'Broker ID', 'Value': broker_id}
],
StartTime=datetime.utcnow() - timedelta(hours=1),
EndTime=datetime.utcnow(),
Period=300,
Statistics=['Average']
)

🔍 Interviewer Notes: Tests load balancing understanding.

❌ Common Mistakes:

  • Not rebalancing leaders after broker changes
  • Ignoring leader skew
  • Not automating leader rebalancing

Q58: What is the FailedProduceRequests metric?​

Technical Answer: FailedProduceRequests counts produce requests that failed due to broker errors, auth failures, or ISR issues. High values indicate cluster instability or misconfiguration.

Code/Config Example:

cloudwatch.put_metric_alarm(
AlarmName='msk-failed-produce-requests',
MetricName='FailedProduceRequests',
Namespace='AWS/Kafka',
Dimensions=[{'Name': 'Cluster Name', 'Value': 'my-cluster'}],
Statistic='Sum',
Period=300,
EvaluationPeriods=2,
Threshold=100,
ComparisonOperator='GreaterThanThreshold'
)

🔍 Interviewer Notes: Tests error monitoring knowledge.

❌ Common Mistakes:

  • Not investigating failure root causes
  • Ignoring transient failures
  • Not correlating with broker health

Q59: How do you monitor MSK cross-AZ traffic?​

Technical Answer: Use the NetworkBytesIn/Out metrics and check traffic between brokers in different AZs. High cross-AZ traffic may indicate unbalanced partition distribution.

Code/Config Example:

# Monitor network traffic
response = cloudwatch.get_metric_statistics(
Namespace='AWS/Kafka',
MetricName='NetworkBytesOut',
Dimensions=[
{'Name': 'Cluster Name', 'Value': 'my-cluster'},
{'Name': 'Broker ID', 'Value': '1'}
],
StartTime=datetime.utcnow() - timedelta(hours=1),
EndTime=datetime.utcnow(),
Period=300,
Statistics=['Average']
)

🔍 Interviewer Notes: Tests network and AZ awareness.

❌ Common Mistakes:

  • Not considering AZ distribution of brokers
  • Ignoring cross-AZ data transfer costs
  • Not balancing partitions across AZs

Q60: How do you export MSK metrics to Prometheus?​

Technical Answer: Use the JMX exporter as a Java agent on brokers or deploy a Prometheus JMX exporter sidecar. MSK Serverless does not support custom metric exporters.

Code/Config Example:

# Prometheus JMX exporter config
jmx_exporter:
startDelaySeconds: 0
lowercaseOutputName: true
rules:
- pattern: "kafka.server<type=BrokerTopicMetrics, name=(MessagesInPerSec|BytesInPerSec)><>OneMinuteRate"
name: "kafka_broker_$1"
type: GAUGE

🔍 Interviewer Notes: Tests custom monitoring integration.

❌ Common Mistakes:

  • Trying to use JMX exporter with MSK Serverless
  • Not securing metric endpoints
  • High scrape interval causing overhead

Q61: How do you scale MSK clusters by adding brokers?​

Technical Answer: Use the UpdateBrokerCount API to add brokers to provisioned MSK clusters. New brokers automatically join the cluster, but partitions are not auto-rebalanced.

Code/Config Example:

kafka = boto3.client('kafka')

response = kafka.update_broker_count(
ClusterArn='arn:aws:kafka:us-east-1:123456789:cluster/my-cluster/abcd1234',
CurrentVersion='ABC123',
TargetNumberOfBrokerNodes=6 # Increase from 3 to 6
)

🔍 Interviewer Notes: Tests scaling operations knowledge.

❌ Common Mistakes:

  • Not rebalancing partitions after adding brokers
  • Scaling too quickly (causes rebalancing storms)
  • Not updating client bootstrap servers

Q62: What is the difference between scaling up and scaling out MSK?​

Technical Answer: Scaling up increases broker instance types (e.g., m5.large to m5.xlarge) for more CPU/memory per broker. Scaling out adds more broker nodes to increase total cluster capacity.

Code/Config Example:

# Scale up (update instance type)
kafka.update_broker_type(
ClusterArn='arn:aws:kafka:us-east-1:123456789:cluster/my-cluster/abcd1234',
CurrentVersion='ABC123',
BrokerNodeGroupInfo={
'InstanceType': 'kafka.m5.xlarge' # Scale up from m5.large
}
)

🔍 Interviewer Notes: Tests scaling strategy understanding.

❌ Common Mistakes:

  • Scaling up when more parallelism is needed (scale out instead)
  • Scaling out when CPU/memory is bottleneck (scale up instead)
  • Not testing performance after scaling

Q63: How do you handle partition rebalancing after adding brokers?​

Technical Answer: Use the Kafka partition reassignment tool to manually move partitions to new brokers. MSK does not auto-rebalance partitions after scaling out.

Code/Config Example:

# Generate partition reassignment plan
kafka-reassign-partitions.sh \
--bootstrap-server broker:9092 \
--topics-to-move-json-file topics.json \
--broker-list "1,2,3,4,5,6" \
--generate

# Execute reassignment
kafka-reassign-partitions.sh \
--bootstrap-server broker:9092 \
--reassignment-json-file reassignment.json \
--execute

🔍 Interviewer Notes: Tests post-scaling operations.

❌ Common Mistakes:

  • Not rebalancing after adding brokers
  • Moving too many partitions at once
  • Not monitoring reassignment progress

Q64: What is MSK elastic scaling?​

Technical Answer: MSK elastic scaling allows automatic scaling of broker count based on cluster load. It is available for provisioned clusters and adjusts broker count within a defined range.

Code/Config Example:

kafka.update_cluster_configuration(
ClusterArn='arn:aws:kafka:us-east-1:123456789:cluster/my-cluster/abcd1234',
ConfigurationInfo={
'ElasticScalingConfiguration': {
'MinimumNumberOfBrokerNodes': 3,
'MaximumNumberOfBrokerNodes': 9
}
}
)

🔍 Interviewer Notes: Tests managed scaling features.

❌ Common Mistakes:

  • Setting min/max too close
  • Not monitoring elastic scaling events
  • Conflicting with manual scaling

Q65: How do you scale MSK Serverless clusters?​

Technical Answer: MSK Serverless scales automatically based on throughput and storage needs. You cannot manually add brokers, but can adjust the maximum throughput and storage limits.

Code/Config Example:

kafka.update_serverless_cluster(
ClusterArn='arn:aws:kafka:us-east-1:123456789:cluster/my-serverless-cluster/abcd1234',
Serverless={
'VpcConfigs': [{'SubnetIds': ['subnet-123'], 'SecurityGroups': ['sg-123']}],
'ClientAuthentication': {'Sasl': {'Iam': {'Enabled': True}}}
}
)

🔍 Interviewer Notes: Tests Serverless scaling knowledge.

❌ Common Mistakes:

  • Trying to manually add brokers to Serverless
  • Not setting throughput limits
  • Confusing Serverless and Provisioned scaling

Q66: How do you remove brokers from MSK clusters?​

Technical Answer: Use the UpdateBrokerCount API to reduce broker count. MSK automatically moves partitions off the removed brokers before terminating them to avoid data loss.

Code/Config Example:

kafka.update_broker_count(
ClusterArn='arn:aws:kafka:us-east-1:123456789:cluster/my-cluster/abcd1234',
CurrentVersion='ABC123',
TargetNumberOfBrokerNodes=3 # Reduce from 6 to 3
)

🔍 Interviewer Notes: Tests scaling down operations.

❌ Common Mistakes:

  • Removing brokers below min ISR requirements
  • Not checking under-replicated partitions first
  • Reducing too quickly causing rebalancing storms

Q67: What is broker graceful shutdown in MSK?​

Technical Answer: MSK brokers shut down gracefully by default, moving leader partitions to other brokers before termination. This minimizes downtime during maintenance or scaling.

Code/Config Example:

# Check broker shutdown status via describe_cluster_operation
response = kafka.describe_cluster_operation(
ClusterOperationArn='arn:aws:kafka:us-east-1:123456789:cluster/my-cluster/operation/abcd1234'
)
print(response['ClusterOperationInfo']['OperationState']) # SHUTTING_DOWN

🔍 Interviewer Notes: Tests broker lifecycle understanding.

❌ Common Mistakes:

  • Forcing broker termination
  • Not monitoring shutdown progress
  • Assuming immediate termination

Q68: How do you handle broker replacement in MSK?​

Technical Answer: MSK automatically replaces failed brokers in provisioned clusters. You can also manually replace brokers by terminating and adding new ones via UpdateBrokerCount.

Code/Config Example:

# Manual broker replacement (terminate failed, add new)
kafka.update_broker_count(
ClusterArn='arn:aws:kafka:us-east-1:123456789:cluster/my-cluster/abcd1234',
CurrentVersion='ABC123',
TargetNumberOfBrokerNodes=3 # Keep count same after replacement
)

🔍 Interviewer Notes: Tests failure handling procedures.

❌ Common Mistakes:

  • Manually intervening too quickly
  • Not verifying ISR before replacement
  • Reducing broker count during replacement

Q69: How do you monitor scaling operations in MSK?​

Technical Answer: Use DescribeClusterOperations API to track scaling progress. CloudWatch metrics show broker count changes and rebalancing activity.

Code/Config Example:

response = kafka.list_cluster_operations(
ClusterArn='arn:aws:kafka:us-east-1:123456789:cluster/my-cluster/abcd1234'
)
for op in response['ClusterOperationInfoList']:
print(f"Operation: {op['OperationType']}, State: {op['OperationState']}")

🔍 Interviewer Notes: Tests operational visibility.

❌ Common Mistakes:

  • Not tracking scaling progress
  • Ignoring failed scaling operations
  • Not setting up operation alerts

Q70: What is the maximum broker count for MSK clusters?​

Technical Answer: MSK supports up to 30 brokers per cluster for provisioned deployments. Serverless clusters auto-scale without a fixed broker limit.

Code/Config Example:

# Check current broker count
response = kafka.describe_cluster(
ClusterArn='arn:aws:kafka:us-east-1:123456789:cluster/my-cluster/abcd1234'
)
print(f"Broker count: {response['ClusterInfo']['NumberOfBrokerNodes']}") # Max 30

🔍 Interviewer Notes: Tests service quotas knowledge.

❌ Common Mistakes:

  • Exceeding 30 broker limit
  • Not requesting quota increases
  • Assuming Serverless has the same limit

Q71: How do you integrate MSK with AWS Lambda?​

Technical Answer: Use MSK as a Lambda event source to process records without managing consumers. Lambda polls MSK topics and invokes your function with batches of records.

Code/Config Example:

lambda_client = boto3.client('lambda')

response = lambda_client.create_event_source_mapping(
EventSourceArn='arn:aws:kafka:us-east-1:123456789:cluster/my-cluster/abcd1234',
FunctionName='process-orders',
SourceAccessConfigurations=[
{'Type': 'SASL_SCRAM', 'URI': 'arn:aws:secretsmanager:us-east-1:123456789:secret:msk-scram'}
],
Topics=['orders'],
BatchSize=100,
MaximumBatchingWindowInSeconds=10
)

🔍 Interviewer Notes: Tests serverless integration knowledge.

❌ Common Mistakes:

  • Not configuring MSK authentication for Lambda
  • Setting batch size too high/low
  • Not handling partial batch failures

Q72: What is the MSK Lambda event source mapping?​

Technical Answer: The event source mapping polls MSK topics, batches records, and invokes Lambda functions. It manages offsets automatically in a DynamoDB table or Lambda's internal store.

Code/Config Example:

# List existing event source mappings
response = lambda_client.list_event_source_mappings(
EventSourceArn='arn:aws:kafka:us-east-1:123456789:cluster/my-cluster/abcd1234'
)

🔍 Interviewer Notes: Tests event-driven architecture.

❌ Common Mistakes:

  • Confusing with Kafka consumer groups
  • Not monitoring mapping errors
  • Not setting up DLQs

Q73: How do you configure Lambda batch processing for MSK?​

Technical Answer: Set BatchSize (max records per invocation) and MaximumBatchingWindowInSeconds to control batching. Larger batches improve throughput but increase latency.

Code/Config Example:

lambda_client.update_event_source_mapping(
UUID='1234abcd-12ab-34cd-56ef-1234567890ab',
BatchSize=500, # Max 1000
MaximumBatchingWindowInSeconds=10 # Max 300
)

🔍 Interviewer Notes: Tests batch optimization.

❌ Common Mistakes:

  • Setting batch size too high (timeout issues)
  • Not matching batch size to function throughput
  • Ignoring batching window latency

Q74: How do you handle MSK poison messages in Lambda?​

Technical Answer: Configure a Dead Letter Queue (DLQ) for the event source mapping. Failed batches are sent to the DLQ after retries are exhausted.

Code/Config Example:

lambda_client.update_event_source_mapping(
UUID='1234abcd-12ab-34cd-56ef-1234567890ab',
DestinationConfig={
'OnFailure': {
'Destination': 'arn:aws:sqs:us-east-1:123456789:msk-dlq'
}
}
)

🔍 Interviewer Notes: Tests error handling in serverless.

❌ Common Mistakes:

  • Not configuring DLQs
  • Not processing DLQ messages
  • Infinite retry loops

Q75: What are Lambda scaling limits for MSK?​

Technical Answer: Lambda scales up to 1000 concurrent invocations per event source mapping. Throughput is limited by batch size, polling interval, and function duration.

Code/Config Example:

# Monitor Lambda concurrency for MSK
cloudwatch.get_metric_statistics(
Namespace='AWS/Lambda',
MetricName='ConcurrentExecutions',
Dimensions=[{'Name': 'FunctionName', 'Value': 'process-orders'}],
StartTime=datetime.utcnow() - timedelta(hours=1),
EndTime=datetime.utcnow(),
Period=300,
Statistics=['Maximum']
)

🔍 Interviewer Notes: Tests scaling limits knowledge.

❌ Common Mistakes:

  • Not considering Lambda concurrency limits
  • Not throttling MSK produce rate
  • Ignoring function timeouts

Q76: How do you configure Lambda for MSK IAM authentication?​

Technical Answer: Grant the Lambda execution role permissions to connect to MSK and read topics. Use IAM authentication in the event source mapping configuration.

Code/Config Example:

# Lambda execution role policy
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["kafka-cluster:Connect", "kafka-cluster:Read"],
"Resource": "arn:aws:kafka:us-east-1:123456789:cluster/my-cluster/*"
}]
}

# Event source mapping with IAM
lambda_client.create_event_source_mapping(
EventSourceArn='arn:aws:kafka:us-east-1:123456789:cluster/my-cluster/abcd1234',
FunctionName='process-orders',
SourceAccessConfigurations=[{'Type': 'IAM', 'URI': ''}],
Topics=['orders']
)

🔍 Interviewer Notes: Tests IAM integration.

❌ Common Mistakes:

  • Missing IAM permissions for Lambda
  • Using wrong auth type in mapping
  • Not testing authentication

Q77: How do you monitor Lambda MSK processing errors?​

Technical Answer: Monitor Lambda error metrics (Errors, Throttles) and event source mapping metrics (AgeOfOldestRecord, IteratorAge). Set alarms for high error rates.

Code/Config Example:

cloudwatch.put_metric_alarm(
AlarmName='lambda-msk-errors',
MetricName='Errors',
Namespace='AWS/Lambda',
Dimensions=[{'Name': 'FunctionName', 'Value': 'process-orders'}],
Threshold=10,
ComparisonOperator='GreaterThanThreshold'
)

🔍 Interviewer Notes: Tests monitoring serverless integrations.

❌ Common Mistakes:

  • Not monitoring iterator age
  • Ignoring DLQ metrics
  • Not alerting on function errors

Q78: What is the maximum batch size for Lambda MSK?​

Technical Answer: Maximum batch size is 1000 records for MSK event source mappings. Batch size should be tuned to your function's processing capacity and timeout.

Code/Config Example:

# Optimal batch size calculation
def calculate_batch_size(function_timeout_ms, avg_processing_ms):
return min(1000, function_timeout_ms // avg_processing_ms)

print(calculate_batch_size(30000, 50)) # 600 records per batch

🔍 Interviewer Notes: Tests batch tuning skills.

❌ Common Mistakes:

  • Setting batch size too high (timeout)
  • Not matching batch size to record size
  • Ignoring function timeout

Q79: How do you retry failed Lambda MSK events?​

Technical Answer: Lambda retries failed batches automatically up to 2 times by default. Configure the maximum retry count via the event source mapping.

Code/Config Example:

lambda_client.update_event_source_mapping(
UUID='1234abcd-12ab-34cd-56ef-1234567890ab',
MaximumRetryAttempts=3 # Default 2, max 10000
)

🔍 Interviewer Notes: Tests retry logic understanding.

❌ Common Mistakes:

  • Not handling idempotent retries
  • Setting retry count too high
  • Not using DLQs for permanent failures

Q80: How do you use Lambda with MSK Serverless?​

Technical Answer: MSK Serverless works with Lambda event source mappings using IAM authentication. Serverless clusters auto-scale to handle Lambda poll throughput.

Code/Config Example:

lambda_client.create_event_source_mapping(
EventSourceArn='arn:aws:kafka:us-east-1:123456789:cluster/my-serverless-cluster/abcd1234',
FunctionName='process-orders',
SourceAccessConfigurations=[{'Type': 'IAM'}],
Topics=['orders']
)

🔍 Interviewer Notes: Tests Serverless integration.

❌ Common Mistakes:

  • Using SCRAM with Serverless
  • Not granting IAM permissions
  • Expecting fixed polling intervals

Q81: How do you create MSK Connect connectors?​

Technical Answer: Use the CreateConnector API to deploy source or sink connectors. Specify connector configuration, capacity, and an IAM role to access external resources like S3 or databases.

Code/Config Example:

kafkaconnect = boto3.client('kafkaconnect')

response = kafkaconnect.create_connector(
connectorName='s3-sink-connector',
kafkaCluster={
'apacheKafkaCluster': {
'bootstrapServers': ','.join(bootstrap_servers),
'vpc': {'subnets': ['subnet-123'], 'securityGroups': ['sg-123']}
}
},
capacity={'autoScaling': {'maxWorkerCount': 4, 'mcuCount': 1, 'minWorkerCount': 1}},
connectorConfiguration={
'connector.class': 'io.confluent.connect.s3.S3SinkConnector',
's3.bucket.name': 'my-kafka-backup',
'topics': 'orders'
},
kafkaConnectVersion='2.7.1',
serviceExecutionRoleArn='arn:aws:iam::123456789:role/MSKConnectRole'
)

🔍 Interviewer Notes: Tests MSK Connect deployment.

❌ Common Mistakes:

  • Not granting IAM role permissions
  • Wrong connector class name
  • Insufficient worker capacity

Q82: What are common MSK Connect source connectors?​

Technical Answer: Common source connectors include JDBC (databases), Debezium (CDC), FileStream, and S3. These pull data from external systems into MSK topics.

Code/Config Example:

# Debezium MySQL CDC source connector config
connector_config = {
'connector.class': 'io.debezium.connector.mysql.MySqlConnector',
'database.hostname': 'mysql.example.com',
'database.port': '3306',
'database.user': 'kafka',
'database.password': '${secretsmanager:mysql-creds}',
'database.server.id': '184054',
'table.include.list': 'orders.*',
'database.history.kafka.bootstrap.servers': ','.join(bootstrap_servers),
'database.history.kafka.topic': 'dbhistory.orders'
}

🔍 Interviewer Notes: Tests data ingestion patterns.

❌ Common Mistakes:

  • Not using Secrets Manager for credentials
  • Missing database history topic
  • Wrong table filters

Q83: What are common MSK Connect sink connectors?​

Technical Answer: Common sink connectors include S3, Elasticsearch, JDBC, Snowflake, and BigQuery. These export MSK topic data to external systems.

Code/Config Example:

# Snowflake sink connector config
connector_config = {
'connector.class': 'com.snowflake.kafka.connector.SnowflakeSinkConnector',
'topics': 'orders',
'snowflake.url.name': 'myaccount.snowflakecomputing.com',
'snowflake.user.name': 'kafka_user',
'snowflake.private.key': '${secretsmanager:snowflake-key}',
'snowflake.database.name': 'KAFKA_DB',
'snowflake.schema.name': 'PUBLIC'
}

🔍 Interviewer Notes: Tests data export patterns.

❌ Common Mistakes:

  • Not configuring error handling
  • Wrong topic list
  • Missing Snowflake stage

Q84: How do you scale MSK Connect workers?​

Technical Answer: Use auto-scaling to adjust worker count based on throughput. Each worker runs connector tasks, so more workers = more parallelism.

Code/Config Example:

kafkaconnect.update_connector(
connectorArn='arn:aws:kafkaconnect:us-east-1:123456789:connector/s3-sink-connector',
capacity={
'autoScaling': {
'maxWorkerCount': 8, # Increase from 4
'mcuCount': 2, # Increase MCU per worker
'minWorkerCount': 2
}
}
)

🔍 Interviewer Notes: Tests scaling connectors.

❌ Common Mistakes:

  • Under-provisioning workers
  • Not monitoring worker utilization
  • Setting max workers too low

Q85: How do you monitor MSK Connect?​

Technical Answer: Monitor connectors via CloudWatch metrics (WorkerUtilization, RecordsIn, RecordsOut) and connector logs in CloudWatch Logs.

Code/Config Example:

cloudwatch.get_metric_statistics(
Namespace='AWS/KafkaConnect',
MetricName='WorkerUtilization',
Dimensions=[{'Name': 'Connector', 'Value': 's3-sink-connector'}],
StartTime=datetime.utcnow() - timedelta(hours=1),
EndTime=datetime.utcnow(),
Period=300,
Statistics=['Average', 'Maximum']
)

🔍 Interviewer Notes: Tests connector monitoring.

❌ Common Mistakes:

  • Not monitoring worker utilization
  • Ignoring failed task metrics
  • Not setting up connector alarms

Q86: How do you handle MSK Connect failures?​

Technical Answer: Configure dead letter queues (DLQs) for connectors and monitor failed record metrics. Restart failed tasks via the RestartConnectorTask API.

Code/Config Example:

# Restart failed connector task
kafkaconnect.restart_connector_task(
connectorArn='arn:aws:kafkaconnect:us-east-1:123456789:connector/s3-sink-connector',
taskId=1
)

🔍 Interviewer Notes: Tests failure handling.

❌ Common Mistakes:

  • Not configuring DLQs
  • Ignoring failed tasks
  • Not investigating root causes

Q87: What is MSK Connect custom connector plugin?​

Technical Answer: Custom plugins let you deploy proprietary or third-party connectors not included in MSK Connect by default. Upload plugin JARs to S3 and reference them in the connector config.

Code/Config Example:

response = kafkaconnect.create_custom_plugin(
contentType='JAR',
location={'s3Location': {'bucketArn': 'arn:aws:s3:::my-plugins', 'fileKey': 'my-connector.jar'}},
name='my-custom-connector'
)

🔍 Interviewer Notes: Tests custom connector deployment.

❌ Common Mistakes:

  • Wrong plugin content type
  • Not granting S3 read permissions
  • Incompatible Kafka Connect version

Q88: How do you configure MSK Connect with IAM?​

Technical Answer: Use IAM authentication for MSK clusters and grant the connector's execution role permissions to read/write topics and access external resources.

Code/Config Example:

# Connector IAM policy
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["kafka-cluster:Connect", "kafka-cluster:Read"],
"Resource": "arn:aws:kafka:us-east-1:123456789:cluster/my-cluster/*"
}]
}

🔍 Interviewer Notes: Tests IAM integration.

❌ Common Mistakes:

  • Missing MSK topic permissions
  • Not granting S3/Secrets Manager access
  • Wrong IAM role for connector

Q89: How do you use MSK Connect with S3?​

Technical Answer: Deploy the Confluent S3 sink connector to export MSK topic data to S3 buckets with configurable format (Parquet, JSON) and partitioning.

Code/Config Example:

connector_config = {
'connector.class': 'io.confluent.connect.s3.S3SinkConnector',
's3.bucket.name': 'my-kafka-backup',
'format.class': 'io.confluent.connect.s3.format.parquet.ParquetFormat',
'partitioner.class': 'io.confluent.connect.storage.partitioner.TimeBasedPartitioner',
'partition.duration.ms': '3600000', # 1 hour partitions
'topics': 'orders,payments'
}

🔍 Interviewer Notes: Tests S3 integration.

❌ Common Mistakes:

  • Wrong S3 bucket permissions
  • Not configuring partitioning
  • Unsupported file format

Q90: How do you update MSK Connect connector config?​

Technical Answer: Use the UpdateConnectorConfig API to modify connector settings. Some changes require connector restart to take effect.

Code/Config Example:

kafkaconnect.update_connector(
connectorArn='arn:aws:kafkaconnect:us-east-1:123456789:connector/s3-sink-connector',
connectorConfiguration={
'flush.size': '2000', # Increase from default
's3.compression.type': 'gzip'
}
)

🔍 Interviewer Notes: Tests connector maintenance.

❌ Common Mistakes:

  • Not restarting after config change
  • Invalid configuration values
  • Not testing changes first

Q91: How do you deploy MSK clusters in multi-VPC?​

Technical Answer: Deploy MSK clusters in a central VPC and use VPC peering, Transit Gateway, or PrivateLink to access from other VPCs. Each VPC needs routes to the MSK subnet CIDRs.

Code/Config Example:

# Create VPC peering connection
ec2 = boto3.client('ec2')
ec2.create_vpc_peering_connection(
VpcId='vpc-123', # Client VPC
PeerVpcId='vpc-456' # MSK VPC
)

🔍 Interviewer Notes: Tests multi-VPC networking.

❌ Common Mistakes:

  • Missing route table entries
  • Not updating security groups
  • Overlapping CIDR blocks

Q92: What is an MSK private cluster?​

Technical Answer: Private MSK clusters have no public endpoints; clients access brokers via VPC private IPs only. This improves security by eliminating internet-accessible endpoints.

Code/Config Example:

# Create private MSK cluster (no public access)
response = kafka.create_cluster(
ClusterName='private-msk-cluster',
KafkaVersion='3.5.1',
NumberOfBrokerNodes=3,
BrokerNodeGroupInfo={
'InstanceType': 'kafka.m5.large',
'ClientSubnets': ['subnet-123', 'subnet-456'],
'SecurityGroups': ['sg-123']
},
ClientAuthentication={'Sasl': {'Iam': {'Enabled': True}}}
)

🔍 Interviewer Notes: Tests private cluster deployment.

❌ Common Mistakes:

  • Trying to access private cluster from internet
  • Not configuring VPC connectivity
  • Missing private DNS resolution

Q93: How do you access MSK from another VPC?​

Technical Answer: Use VPC peering or Transit Gateway to route traffic between the client VPC and MSK VPC. Update security groups to allow client VPC CIDR on broker ports (9098 for TLS).

Code/Config Example:

# Update MSK security group to allow client VPC
ec2.authorize_security_group_ingress(
GroupId='sg-123', # MSK security group
IpPermissions=[{
'IpProtocol': 'tcp',
'FromPort': 9098,
'ToPort': 9098,
'IpRanges': [{'CidrIp': '10.0.1.0/24'}] # Client VPC CIDR
}]
)

🔍 Interviewer Notes: Tests cross-VPC access.

❌ Common Mistakes:

  • Not updating security groups
  • Missing route table entries
  • Wrong broker port for auth type

Q94: How do you use VPC peering with MSK?​

Technical Answer: Create a VPC peering connection between client and MSK VPCs, update route tables on both sides, and ensure no overlapping CIDRs.

Code/Config Example:

# Update route table for client VPC
ec2.create_route(
RouteTableId='rtb-123', # Client VPC route table
DestinationCidrBlock='10.0.2.0/24', # MSK VPC CIDR
VpcPeeringConnectionId='pcx-123'
)

🔍 Interviewer Notes: Tests VPC peering configuration.

❌ Common Mistakes:

  • Overlapping CIDR blocks
  • Not updating both route tables
  • Missing peering connection acceptance

Q95: How do you use Transit Gateway with MSK?​

Technical Answer: Attach both client and MSK VPCs to a Transit Gateway, propagate routes, and update security groups to allow cross-VPC traffic.

Code/Config Example:

# Create Transit Gateway attachment for MSK VPC
ec2.create_transit_gateway_vpc_attachment(
TransitGatewayId='tgw-123',
VpcId='vpc-456', # MSK VPC
SubnetIds=['subnet-123', 'subnet-456']
)

🔍 Interviewer Notes: Tests Transit Gateway usage.

❌ Common Mistakes:

  • Not propagating routes
  • Missing attachments for both VPCs
  • Incorrect security group rules

Q96: What is an MSK VPC endpoint?​

Technical Answer: MSK VPC endpoints (PrivateLink) let you access MSK clusters from other VPCs or accounts without VPC peering. Endpoints are created in the client VPC pointing to the MSK cluster.

Code/Config Example:

# Create VPC endpoint for MSK (PrivateLink)
ec2.create_vpc_endpoint(
VpcId='vpc-123', # Client VPC
ServiceName='com.amazonaws.us-east-1.kafka-cluster',
SubnetIds=['subnet-123'],
SecurityGroupIds=['sg-123']
)

🔍 Interviewer Notes: Tests PrivateLink knowledge.

❌ Common Mistakes:

  • Not using correct service name
  • Missing endpoint security groups
  • Wrong subnet selection

Q97: How do you secure MSK cross-VPC access?​

Technical Answer: Use security groups to restrict broker access to known VPC CIDRs, enable IAM authentication, and encrypt traffic with TLS.

Code/Config Example:

# Restrictive security group for MSK
{
"IpProtocol": "tcp",
"FromPort": 9098,
"ToPort": 9098,
"SourceSecurityGroupId": "sg-client" # Only allow client SG
}

🔍 Interviewer Notes: Tests cross-VPC security.

❌ Common Mistakes:

  • Overly permissive security groups
  • Not using IAM authentication
  • Missing TLS encryption

Q98: How do you deploy MSK in shared VPCs?​

Technical Answer: Deploy MSK in a shared services VPC with separate subnets per team. Use security groups and IAM policies to enforce isolation between tenants.

Code/Config Example:

# Shared VPC MSK subnet for team A
{
"SubnetId": "subnet-team-a",
"SecurityGroupId": "sg-team-a", # Isolated SG for team A
"InstanceType": "kafka.m5.large"
}

🔍 Interviewer Notes: Tests multi-tenant patterns.

❌ Common Mistakes:

  • Shared security groups between teams
  • Not isolating subnets
  • Weak IAM policies

Q99: How do you test cross-VPC MSK connectivity?​

Technical Answer: Launch a test EC2 instance in the client VPC, install Kafka tools, and run kafka-console-producer/consumer to verify connectivity.

Code/Config Example:

# Test MSK connectivity from client VPC
kafka-console-consumer.sh \
--bootstrap-server $BS:9098 \
--topic test \
--consumer.config client.properties \
--from-beginning

🔍 Interviewer Notes: Tests connectivity troubleshooting.

❌ Common Mistakes:

  • Not testing from same VPC
  • Wrong security group config
  • Missing IAM permissions

Q100: How do you monitor cross-VPC MSK traffic?​

Technical Answer: Use VPC Flow Logs on both client and MSK VPCs to track traffic. Monitor MSK CloudWatch metrics for cross-AZ traffic patterns.

Code/Config Example:

# Enable VPC Flow Logs for MSK VPC
ec2.create_flow_logs(
ResourceIds=['vpc-456'], # MSK VPC
ResourceType='VPC',
TrafficType='ALL',
LogDestinationType='cloud-watch-logs',
LogGroupName='/aws/vpc/msk-vpc'
)

🔍 Interviewer Notes: Tests cross-VPC monitoring.

❌ Common Mistakes:

  • Not enabling Flow Logs
  • Ignoring cross-AZ traffic costs
  • Not monitoring broker security group rejects

2. Intermediate Topics​


2. Intermediate Topics​

Q101: How do you monitor and optimize MSK performance?​

Technical Answer:

CloudWatch Metrics:

import boto3

cloudwatch = boto3.client('cloudwatch')

# Key metrics
metrics = cloudwatch.get_metric_statistics(
Namespace='AWS/Kafka',
MetricName='BytesInPerSec',
Dimensions=[
{'Name': 'Cluster Name', 'Value': 'my-cluster'},
{'Name': 'Broker ID', 'Value': '1'}
],
StartTime=datetime.utcnow() - timedelta(hours=1),
EndTime=datetime.utcnow(),
Period=300,
Statistics=['Average', 'Maximum']
)

# Monitor:
# - BytesInPerSec / BytesOutPerSec
# - MessagesInPerSec
# - CpuUtilization
# - MemoryFree
# - PartitionCount
# - LeaderCount

🔍 Interviewer Notes: Testing monitoring and performance tuning.


Q102: How do you secure MSK clusters?​

Technical Answer:

Encryption:

  • At rest: KMS encryption for EBS volumes
  • In transit: TLS 1.2 for client-broker and inter-broker

Authentication:

  • IAM authentication (recommended)
  • SASL/SCRAM with Secrets Manager
  • Mutual TLS (mTLS)

Authorization:

  • IAM policies for IAM auth
  • ACLs for other auth methods
# IAM policy for MSK access
msk_policy = {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": [
"kafka-cluster:Connect",
"kafka-cluster:DescribeCluster"
],
"Resource": "arn:aws:kafka:us-east-1:123456789:cluster/my-cluster/*"
}, {
"Effect": "Allow",
"Action": ["kafka-cluster:*Topic*"],
"Resource": "arn:aws:kafka:us-east-1:123456789:topic/my-cluster/*/orders"
}]
}

🔍 Interviewer Notes: Testing security best practices.


Q103: How do you handle MSK rebalancing?​

Technical Answer:

Cooperative Rebalancing:

consumer = KafkaConsumer(
'orders',
bootstrap_servers=bootstrap_servers,
group_id='order-processors',
partition_assignment_strategy='org.apache.kafka.clients.consumer.CooperativeStickyAssignor',
# More stable than eager rebalancing
)

Rebalance Listeners:

from kafka import ConsumerRebalanceListener

class RebalanceListener(ConsumerRebalanceListener):
def on_partitions_revoked(self, revoked):
print(f"Revoked: {revoked}")
# Commit offsets before losing partitions
consumer.commit_sync()

def on_partitions_assigned(self, assigned):
print(f"Assigned: {assigned}")
# Seek to correct position if needed

consumer.subscribe(['orders'], listener=RebalanceListener())

🔍 Interviewer Notes: Testing consumer group understanding.


Q104: How do you tune MSK producer batching?​

Technical Answer: Increase batch_size (16KB-128KB) and linger_ms (5-100ms) to send larger batches, reducing network overhead and increasing throughput. Balance batch size with latency requirements.

Code/Config Example:

producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
batch_size=65536, # 64KB batches
linger_ms=50, # Wait up to 50ms to batch
compression_type='snappy'
)

🔍 Interviewer Notes: Tests producer throughput tuning.

❌ Common Mistakes:

  • Batch size too small (high overhead)
  • Linger_ms too high (increased latency)
  • Not matching batch size to message size

Q105: What is the optimal linger.ms setting for MSK?​

Technical Answer: Optimal linger.ms is 5-100ms: lower for low latency, higher for high throughput. Test with your workload to find the sweet spot between latency and batch efficiency.

Code/Config Example:

# Low latency setting
low_latency_producer = KafkaProducer(linger_ms=5)

# High throughput setting
high_throughput_producer = KafkaProducer(linger_ms=100)

🔍 Interviewer Notes: Tests latency vs throughput tradeoffs.

❌ Common Mistakes:

  • Setting linger_ms=0 (no batching)
  • Setting linger_ms too high for latency-sensitive apps
  • Not testing with real workload

Q106: How do you tune MSK producer compression?​

Technical Answer: Use snappy for balanced speed/ratio, lz4 for faster decompression, zstd for best compression ratio, or gzip for compatibility. Compression reduces network and storage costs.

Code/Config Example:

producer = KafkaProducer(
compression_type='lz4' # Faster decompression than snappy
)

🔍 Interviewer Notes: Tests compression strategy.

❌ Common Mistakes:

  • Not using compression (high network costs)
  • Using gzip for high throughput (slow)
  • Not matching compression to broker support

Q107: How do you tune MSK consumer max.poll.records?​

Technical Answer: Set max.poll.records to 500-2000 depending on processing time per record. Larger batches improve throughput but increase risk of max.poll.interval.ms timeout.

Code/Config Example:

consumer = KafkaConsumer(
max_poll_records=1000, # Process 1000 records per poll
max_poll_interval_ms=300000 # 5 minutes to process batch
)

🔍 Interviewer Notes: Tests consumer batch tuning.

❌ Common Mistakes:

  • Setting max_poll_records too high (timeout)
  • Not adjusting max_poll.interval.ms
  • Not matching batch size to processing speed

Q108: What is the max.poll.interval.ms setting for MSK consumers?​

Technical Answer: max.poll.interval.ms is the maximum time between poll() calls before the consumer is kicked out of the group. Set to 5-10x your maximum batch processing time.

Code/Config Example:

# For long processing batches
consumer = KafkaConsumer(
max_poll_records=2000,
max_poll_interval_ms=600000 # 10 minutes
)

🔍 Interviewer Notes: Tests consumer rebalancing prevention.

❌ Common Mistakes:

  • Setting too low (frequent rebalances)
  • Setting too high (slow failure detection)
  • Not matching to processing time

Q109: How do you tune MSK partition count for throughput?​

Technical Answer: Calculate partitions as (target throughput) / (per-partition throughput: ~10MB/s producer, ~10MB/s consumer). Ensure partitions >= number of consumers.

Code/Config Example:

def calculate_partitions(target_mbps, consumer_count):
partitions = max(target_mbps // 10, consumer_count)
return int(partitions)

print(calculate_partitions(100, 20)) # 10 partitions? No, 100/10=10, max(10,20)=20

🔍 Interviewer Notes: Tests capacity planning.

❌ Common Mistakes:

  • Too few partitions (bottleneck)
  • Too many partitions (rebalancing overhead)
  • Not matching to consumer count

Q110: What is MSK broker instance type selection?​

Technical Answer: Choose instance types based on throughput, storage, and network needs: m5 for balanced, r5 for memory-heavy (Kafka Streams), i3 for high IOPS (fast storage).

Code/Config Example:

# High throughput cluster
response = kafka.create_cluster(
BrokerNodeGroupInfo={
'InstanceType': 'kafka.m5.2xlarge' # 8 vCPU, 32GB RAM
}
)

🔍 Interviewer Notes: Tests instance selection.

❌ Common Mistakes:

  • Under-provisioning CPU/memory
  • Not considering network throughput
  • Using i3 for low IOPS workloads

Q111: How do you monitor MSK throughput throttling?​

Technical Answer: Monitor BytesInPerSec/BytesOutPerSec against broker network limits. Alert when throughput approaches instance network capacity.

Code/Config Example:

# Alarm for high throughput
cloudwatch.put_metric_alarm(
AlarmName='msk-high-throughput',
MetricName='BytesInPerSec',
Threshold=100000000, # 100MB/s (m5.large limit ~10Gbps)
ComparisonOperator='GreaterThanThreshold'
)

🔍 Interviewer Notes: Tests throughput monitoring.

❌ Common Mistakes:

  • Not knowing instance network limits
  • Ignoring sustained high throughput
  • Not scaling when throttled

Q112: How do you tune MSK socket buffers?​

Technical Answer: Increase send.buffer.bytes and receive.buffer.bytes for high throughput networks. Set to 1MB-4MB for 10Gbps networks.

Code/Config Example:

producer = KafkaProducer(
send_buffer_bytes=1048576, # 1MB send buffer
receive_buffer_bytes=1048576 # 1MB receive buffer
)

🔍 Interviewer Notes: Tests network tuning.

❌ Common Mistakes:

  • Using default buffers (small for high throughput)
  • Setting too high (memory waste)
  • Not matching to network MTU

Q113: What is the MSK producer acks setting tradeoff?​

Technical Answer: acks=0 (lowest latency, no durability), acks=1 (leader ack, medium durability), acks=all (all ISRs, highest durability, higher latency). Choose based on data criticality.

Code/Config Example:

# High durability
durable_producer = KafkaProducer(acks='all', min_insync_replicas=2)

# Low latency
fast_producer = KafkaProducer(acks='1')

🔍 Interviewer Notes: Tests durability vs latency tradeoffs.

❌ Common Mistakes:

  • Using acks=0 for critical data
  • Using acks=all for low latency apps
  • Not matching to min.insync.replicas

Q114: How do you tune MSK consumer fetch size?​

Technical Answer: Set fetch.min.bytes (1-10KB) to wait for enough data before returning, and fetch.max.bytes (10MB-50MB) to limit response size per fetch.

Code/Config Example:

consumer = KafkaConsumer(
fetch_min_bytes=10240, # Wait for 10KB
fetch_max_bytes=52428800, # Max 50MB per fetch
max_partition_fetch_bytes=1048576 # 1MB per partition
)

🔍 Interviewer Notes: Tests fetch tuning.

❌ Common Mistakes:

  • fetch_min_bytes too high (long latency)
  • fetch_max_bytes too small (many fetches)
  • Not adjusting per-partition limits

Q115: What is MSK broker EBS volume tuning?​

Technical Answer: Use gp3 volumes for adjustable IOPS (up to 16000) and throughput (up to 1000MB/s). Size volumes for 2-3x your retention needs.

Code/Config Example:

response = kafka.create_cluster(
BrokerNodeGroupInfo={
'StorageInfo': {
'EbsStorageInfo': {
'VolumeSize': 2000, # 2TB
'ProvisionedIops': 10000
}
}
}
)

🔍 Interviewer Notes: Tests storage tuning.

❌ Common Mistakes:

  • Using gp2 (fixed IOPS)
  • Under-sizing volumes
  • Not monitoring disk usage

Q116: How do you use MSK compression for network optimization?​

Technical Answer: Enable producer compression (snappy, lz4) to reduce network traffic by 50-80%. Brokers forward compressed data without re-compression if compression.type=producer.

Code/Config Example:

# Topic with producer compression
admin_client.create_topics([NewTopic(
name='compressed-events',
num_partitions=12,
topic_configs={'compression.type': 'producer'}
)])

🔍 Interviewer Notes: Tests network cost optimization.

❌ Common Mistakes:

  • Not enabling topic-level compression
  • Using high-overhead compression (gzip)
  • Not testing compression ratio

Q117: What is the MSK producer retries setting?​

Technical Answer: Set retries=5-10 for transient failures (leader not available, network blips). Use retry_backoff_ms=100-1000 to avoid overwhelming brokers.

Code/Config Example:

producer = KafkaProducer(
retries=10,
retry_backoff_ms=500,
enable_idempotence=True # Required for safe retries
)

🔍 Interviewer Notes: Tests retry configuration.

❌ Common Mistakes:

  • Retries=0 (lost messages)
  • Not using idempotence with retries
  • Retry backoff too short (thundering herd)

Q118: How do you tune MSK for low latency?​

Technical Answer: Set linger_ms=0-5, batch_size=small, acks=1, and use low-latency instance types. Avoid compression for latency-critical workloads.

Code/Config Example:

low_latency_producer = KafkaProducer(
linger_ms=0, # No batching wait
batch_size=8192, # Small batches
acks='1', # Leader ack only
compression_type=None # No compression overhead
)

🔍 Interviewer Notes: Tests latency optimization.

❌ Common Mistakes:

  • Using high linger_ms
  • Large batch sizes
  • Compression adding overhead

Q119: How do you measure MSK end-to-end latency?​

Technical Answer: Add timestamps to messages on produce, compare to processing time on consume. Use Kafka's built-in produce/consume timestamps for rough estimates.

Code/Config Example:

# Produce with timestamp
producer.send('orders', value=order, timestamp_ms=int(time.time()*1000))

# Consume and calculate latency
for msg in consumer:
latency = time.time()*1000 - msg.timestamp
print(f"Latency: {latency}ms")

🔍 Interviewer Notes: Tests latency measurement.

❌ Common Mistakes:

  • Not including clock skew
  • Measuring only producer/consumer side
  • Not tracking p99 latency

Q120: How do you tune MSK for high fanout?​

Technical Answer: Increase consumer count (up to partition count), use separate consumer groups per downstream system, and tune fetch sizes for many concurrent reads.

Code/Config Example:

# Multiple consumer groups for fanout
consumer1 = KafkaConsumer(group_id='analytics', ...)
consumer2 = KafkaConsumer(group_id='notifications', ...)

🔍 Interviewer Notes: Tests fanout patterns.

❌ Common Mistakes:

  • More consumers than partitions
  • Not using separate groups
  • High overhead from many groups

Q121: How do you implement MSK disaster recovery?​

Technical Answer: Use MirrorMaker 2 for cross-region replication, take regular EBS snapshots, and maintain a standby cluster in another region with the same configuration.

Code/Config Example:

# MirrorMaker 2 configuration
clusters = primary.boostrap.servers = b-1.primary.kafka.us-east-1.amazonaws.com:9098
clusters = dr.boostrap.servers = b-1.dr.kafka.us-west-2.amazonaws.com:9098
primary->dr.enabled = true

🔍 Interviewer Notes: Tests DR planning.

❌ Common Mistakes:

  • Not testing failover
  • Single region deployment
  • No cross-region replication

Q122: What is MSK cross-region replication?​

Technical Answer: Replicate topics between MSK clusters in different regions using MirrorMaker 2 or Kafka Streams. Replication is asynchronous with eventual consistency.

Code/Config Example:

# MirrorMaker 2 topic replication policy
replication_policy_class=org.apache.kafka.connect.mirror.IdentityReplicationPolicy
topics=.* # Replicate all topics

🔍 Interviewer Notes: Tests multi-region patterns.

❌ Common Mistakes:

  • Not filtering sensitive topics
  • High cross-region data transfer costs
  • Not monitoring replication lag

Q123: How do you handle MSK cluster failure?​

Technical Answer: MSK automatically recovers failed brokers. For full cluster failure, failover to a DR cluster in another region using DNS switching or client reconfiguration.

Code/Config Example:

# Failover client to DR cluster
bootstrap_servers = [
'b-1.dr-cluster.kafka.us-west-2.amazonaws.com:9098'
]

🔍 Interviewer Notes: Tests failure recovery.

❌ Common Mistakes:

  • Not having a DR cluster
  • Manual failover delay
  • Not testing failover regularly

Q124: What is MSK replica placement?​

Technical Answer: MSK places replicas across multiple AZs by default for high availability. Ensure your VPC has subnets in 3+ AZs for full replication benefits.

Code/Config Example:

# 3 AZs for replica placement
response = kafka.create_cluster(
BrokerNodeGroupInfo={
'ClientSubnets': ['subnet-az1', 'subnet-az2', 'subnet-az3']
}
)

🔍 Interviewer Notes: Tests AZ awareness.

❌ Common Mistakes:

  • Single AZ deployment
  • Not enough AZs for RF=3
  • Uneven AZ subnet capacity

Q125: How do you backup MSK data?​

Technical Answer: Use MSK Connect to export topics to S3, enable S3 replication for backups, or take EBS volume snapshots of broker nodes (not officially supported, use Connect instead).

Code/Config Example:

# S3 backup connector
connector_config = {
'connector.class': 'io.confluent.connect.s3.S3SinkConnector',
'topics': 'orders,payments',
's3.bucket.name': 'kafka-backup'
}

🔍 Interviewer Notes: Tests backup strategies.

❌ Common Mistakes:

  • No backup strategy
  • Not testing restore
  • Relying only on replication

Q126: How do you restore MSK data from backup?​

Technical Answer: Create a new MSK cluster, deploy an S3 source connector to read backup data back into topics, or replay data from a DR cluster.

Code/Config Example:

# S3 source connector to restore
connector_config = {
'connector.class': 'io.confluent.connect.s3.S3SourceConnector',
's3.bucket.name': 'kafka-backup',
'topics': 'orders'
}

🔍 Interviewer Notes: Tests restore procedures.

❌ Common Mistakes:

  • Not testing restore
  • Data format mismatch
  • Duplicate messages on restore

Q127: What is MSK multi-AZ deployment?​

Technical Answer: Deploy MSK brokers across 2-3 availability zones for high availability. Multi-AZ ensures the cluster survives an AZ failure without downtime.

Code/Config Example:

# Multi-AZ deployment
response = kafka.create_cluster(
BrokerNodeGroupInfo={
'ClientSubnets': ['subnet-az1', 'subnet-az2', 'subnet-az3']
}
)

🔍 Interviewer Notes: Tests HA deployment.

❌ Common Mistakes:

  • Single AZ deployment
  • Not spreading partitions across AZs
  • Uneven AZ capacity

Q128: How do you handle MSK split-brain scenarios?​

Technical Answer: MSK uses ZooKeeper/KRaft quorum to prevent split-brain. Disable unclean leader election to avoid data loss when quorum is lost.

Code/Config Example:

# Disable unclean leader election
admin_client.alter_configs([ConfigResource(
ConfigResourceType.TOPIC, 'orders',
configs={'unclean.leader.election.enable': 'false'}
)])

🔍 Interviewer Notes: Tests split-brain prevention.

❌ Common Mistakes:

  • Enabling unclean leader election
  • Not monitoring quorum health
  • Ignoring AZ failures

Q129: How do you test MSK disaster recovery?​

Technical Answer: Regularly test failover to DR clusters, validate data consistency between primary and replica, and measure recovery time objectives (RTO).

Code/Config Example:

# Validate replica data
primary_messages = consume_all('primary-orders')
dr_messages = consume_all('dr-orders')
assert len(primary_messages) <= len(dr_messages)

🔍 Interviewer Notes: Tests DR testing.

❌ Common Mistakes:

  • No regular DR tests
  • Not measuring RTO/RPO
  • Ignoring data consistency

Q130: What is RTO and RPO for MSK?​

Technical Answer: RTO (Recovery Time Objective) is maximum acceptable downtime (e.g., 1 hour). RPO (Recovery Point Objective) is maximum acceptable data loss (e.g., 5 minutes).

Code/Config Example:

# RPO: 5 minutes of data loss
# Set retention.ms=300000 on critical topics
admin_client.alter_configs([ConfigResource(
ConfigResourceType.TOPIC, 'orders',
configs={'retention.ms': '300000'}
)])

🔍 Interviewer Notes: Tests disaster recovery metrics.

❌ Common Mistakes:

  • Not defining RTO/RPO
  • RPO longer than retention
  • Not monitoring replication lag

Q131: How do you replicate MSK topics to another account?​

Technical Answer: Use MirrorMaker 2 with IAM roles for cross-account access, or MSK Connect with cross-account S3 buckets for data replication.

Code/Config Example:

# Cross-account IAM role for replication
{
"Effect": "Allow",
"Action": ["kafka-cluster:Connect", "kafka-cluster:Read"],
"Resource": "arn:aws:kafka:us-west-2:987654321:cluster/dr-cluster/*"
}

🔍 Interviewer Notes: Tests cross-account patterns.

❌ Common Mistakes:

  • Not setting up cross-account IAM
  • Data residency compliance issues
  • High cross-account data costs

Q132: How do you handle MSK replication lag?​

Technical Answer: Monitor replication lag between primary and DR clusters. Increase DR cluster capacity, optimize MirrorMaker 2 workers, or reduce topic retention to control lag.

Code/Config Example:

# Monitor replication lag
cloudwatch.get_metric_statistics(
Namespace='AWS/Kafka',
MetricName='ReplicationLag',
Dimensions=[{'Name': 'Cluster Name', 'Value': 'dr-cluster'}]
)

🔍 Interviewer Notes: Tests replication monitoring.

❌ Common Mistakes:

  • Ignoring high replication lag
  • Not scaling DR cluster
  • Not optimizing MirrorMaker workers

Q133: What is MSK buried data recovery?​

Technical Answer: If data is deleted from primary, it may still exist in DR clusters or S3 backups. Restore from backups or DR clusters before retention expires.

Code/Config Example:

# Restore deleted topic from S3 backup
# Deploy S3 source connector to re-ingest data

🔍 Interviewer Notes: Tests data recovery.

❌ Common Mistakes:

  • Not having backups
  • Retention too short for recovery
  • Not testing recovery

Q134: How do you automate MSK DR failover?​

Technical Answer: Use Route53 health checks to detect primary cluster failure, then automatically switch clients to DR cluster via DNS or configuration management.

Code/Config Example:

# Route53 failover record
route53.change_resource_record_sets(
ChangeBatch={
'Changes': [{
'Action': 'UPSERT',
'ResourceRecordSet': {
'Name': 'kafka.example.com',
'Type': 'CNAME',
'Failover': 'PRIMARY',
'ResourceRecords': [{'Value': 'b-1.primary.kafka.amazonaws.com'}]
}
}]
}
)

🔍 Interviewer Notes: Tests automated failover.

❌ Common Mistakes:

  • Manual failover delay
  • Not testing automation
  • DNS caching delaying failover

Q135: How do you size MSK DR clusters?​

Technical Answer: DR clusters should match primary cluster capacity for 1:1 failover, or be smaller for warm standby (scale out during failover). Consider cross-region data transfer costs.

Code/Config Example:

# Warm standby DR cluster (smaller)
response = kafka.create_cluster(
NumberOfBrokerNodes=2, # Smaller than primary (3)
BrokerNodeGroupInfo={'InstanceType': 'kafka.m5.large'}
)

🔍 Interviewer Notes: Tests DR sizing.

❌ Common Mistakes:

  • Over-provisioning DR (high cost)
  • Under-provisioning (can't handle failover)
  • Not considering data transfer costs

Q136: What is MSK backup retention policy?​

Technical Answer: Set backup retention longer than primary topic retention (e.g., 1 year for backups vs 7 days for primary). Use S3 lifecycle policies to archive old backups to Glacier.

Code/Config Example:

# S3 lifecycle policy for backups
s3.put_bucket_lifecycle_configuration(
Bucket='kafka-backup',
LifecycleConfiguration={
'Rules': [{
'ID': 'archive-old-backups',
'Status': 'Enabled',
'Transitions': [{'Days': 30, 'StorageClass': 'GLACIER'}]
}]
}
)

🔍 Interviewer Notes: Tests backup retention.

❌ Common Mistakes:

  • Backup retention too short
  • High S3 storage costs
  • Not archiving old backups

Q137: How do you encrypt MSK cross-region replication?​

Technical Answer: MSK encrypts data in transit by default between regions. Use KMS CMKs for at-rest encryption in both regions, and encrypt S3 backups with KMS.

Code/Config Example:

# DR cluster with CMK encryption
response = kafka.create_cluster(
EncryptionInfo={
'EncryptionAtRest': {'DataVolumeKMSKeyId': 'dr-cmk-arn'}
}
)

🔍 Interviewer Notes: Tests cross-region encryption.

❌ Common Mistakes:

  • Not encrypting DR clusters
  • Different KMS keys without cross-account access
  • Unencrypted backups

Q138: How do you monitor MSK DR readiness?​

Technical Answer: Monitor replication lag, DR cluster health, backup success rates, and regular DR test results. Set alerts for any DR component failure.

Code/Config Example:

# Alarm for DR cluster failure
cloudwatch.put_metric_alarm(
AlarmName='dr-cluster-down',
MetricName='ClusterStatus',
Dimensions=[{'Name': 'Cluster Name', 'Value': 'dr-cluster'}]
)

🔍 Interviewer Notes: Tests DR monitoring.

❌ Common Mistakes:

  • Not monitoring DR clusters
  • No DR readiness alerts
  • Ignoring backup failures

Q139: What is MSK active-active deployment?​

Technical Answer: Deploy two active MSK clusters in different regions, each handling local traffic and replicating to the other. Requires conflict resolution for conflicting writes.

Code/Config Example:

# Active-active MirrorMaker 2 config
primary->dr.enabled = true
dr->primary.enabled = true

🔍 Interviewer Notes: Tests multi-active patterns.

❌ Common Mistakes:

  • Write conflicts without resolution
  • High cross-region replication costs
  • Not testing conflict scenarios

Q140: How do you handle MSK schema changes in DR?​

Technical Answer: Use Schema Registry in each region, or replicate Schema Registry data between regions. Ensure schema compatibility is maintained across primary and DR.

Code/Config Example:

# Replicate Schema Registry with MirrorMaker 2
topics=_schemas # Replicate Schema Registry topic

🔍 Interviewer Notes: Tests schema DR.

❌ Common Mistakes:

  • Schema mismatch between regions
  • Not replicating schemas
  • Breaking schema compatibility

Q141: What are MSK security best practices?​

Technical Answer: Follow least privilege for IAM, encrypt data at rest/in transit, deploy in private VPCs, enable audit logging, and use TLS 1.2+ for all connections.

Code/Config Example:

# Least privilege IAM policy
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["kafka-cluster:Connect"],
"Resource": "arn:aws:kafka:us-east-1:123456789:cluster/my-cluster/*"
}]
}

🔍 Interviewer Notes: Tests holistic security knowledge.

❌ Common Mistakes:

  • Overly permissive IAM policies
  • Not encrypting data at rest
  • Publicly accessible clusters

Q142: How do you use IAM roles for MSK access?​

Technical Answer: Assign IAM roles to producers/consumers, MSK Connect, and Lambda. Use IAM authentication (AWS_MSK_IAM) to verify role permissions.

Code/Config Example:

# Producer with IAM role
producer = KafkaProducer(
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM',
sasl_aws_msk_iam_auth_token_provider=get_auth_token
)

🔍 Interviewer Notes: Tests IAM integration.

❌ Common Mistakes:

  • Not granting IAM permissions
  • Using wrong auth mechanism
  • Not rotating role credentials

Q143: How do you restrict MSK cluster access by IP?​

Technical Answer: Use security groups to restrict broker port access to specific IP ranges. For on-premises access, use NAT Gateway or VPN with allowed CIDR blocks.

Code/Config Example:

# Security group ingress rule
{
"IpProtocol": "tcp",
"FromPort": 9098,
"ToPort": 9098,
"IpRanges": [{"CidrIp": "203.0.113.0/24"}] # Office IP range
}

🔍 Interviewer Notes: Tests network security.

❌ Common Mistakes:

  • Allowing 0.0.0.0/0 in security groups
  • Not updating IP ranges
  • Missing egress rules

Q144: What is MSK VPC endpoint security?​

Technical Answer: MSK VPC endpoints (PrivateLink) restrict cluster access to specific VPCs/accounts. Endpoints eliminate internet access to brokers.

Code/Config Example:

# Restrictive endpoint policy
{
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": "arn:aws:iam::123456789:root"},
"Action": "kafka-cluster:Connect",
"Resource": "*"
}]
}

🔍 Interviewer Notes: Tests PrivateLink security.

❌ Common Mistakes:

  • Overly permissive endpoint policies
  • Not using endpoints for cross-account access
  • Missing endpoint security groups

Q145: How do you use Cognito with MSK?​

Technical Answer: Use Cognito identity pools to grant temporary AWS credentials for MSK IAM authentication. Map Cognito users/groups to IAM roles with MSK permissions.

Code/Config Example:

# Cognito identity pool IAM role
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["kafka-cluster:Connect", "kafka-cluster:Read"],
"Resource": "arn:aws:kafka:us-east-1:123456789:cluster/my-cluster/*"
}]
}

🔍 Interviewer Notes: Tests Cognito integration.

❌ Common Mistakes:

  • Not mapping Cognito groups to roles
  • Overly permissive roles
  • Not using identity pools

Q146: What is MSK TLS certificate rotation?​

Technical Answer: MSK manages broker TLS certificate rotation automatically. For client certificates (mTLS), rotate them manually before expiration and update clients.

Code/Config Example:

# Check certificate expiration
import ssl, socket
cert = ssl.get_server_certificate(('b-1.my-cluster.kafka.us-east-1.amazonaws.com', 9098))

🔍 Interviewer Notes: Tests certificate management.

❌ Common Mistakes:

  • Not rotating client certificates
  • Expired certificates causing outages
  • Not automating rotation

Q147: How do you enable MSK audit logging?​

Technical Answer: Enable audit logging to CloudWatch Logs to track cluster operations (create topic, delete topic, modify config). Audit logs help with compliance.

Code/Config Example:

kafka.update_cluster_configuration(
ClusterArn='arn:aws:kafka:us-east-1:123456789:cluster/my-cluster/abcd1234',
ConfigurationInfo={
'AuditLogging': {'Enabled': True}
}
)

🔍 Interviewer Notes: Tests audit logging.

❌ Common Mistakes:

  • Not enabling audit logs
  • Not monitoring audit logs
  • High CloudWatch Logs costs

Q148: How do you use MSK with Secrets Manager?​

Technical Answer: Store SASL/SCRAM credentials or client certificates in Secrets Manager. MSK Connect and Lambda can reference secrets directly.

Code/Config Example:

# Reference secret in MSK Connect
connector_configuration={
'sasl.jaas.config': 'org.apache.kafka.common.security.scram.ScramLoginModule required username="${secretsmanager:msk-scram:username}" password="${secretsmanager:msk-scram:password}";'
}

🔍 Interviewer Notes: Tests secrets management.

❌ Common Mistakes:

  • Hardcoding credentials
  • Not rotating secrets
  • Missing IAM permissions for secrets

Q149: What is MSK network segmentation?​

Technical Answer: Deploy MSK in isolated subnets, use separate security groups per tier, and restrict subnet CIDR ranges. Avoid deploying MSK in public subnets.

Code/Config Example:

# Isolated MSK subnet
{
"SubnetId": "subnet-msk-private",
"CidrBlock": "10.0.3.0/24",
"Tags": [{"Key": "Tier", "Value": "data"}]
}

🔍 Interviewer Notes: Tests network design.

❌ Common Mistakes:

  • Public subnet deployment
  • Shared security groups
  • Overlapping CIDR blocks

Q150: How do you implement least privilege for MSK?​

Technical Answer: Grant only required permissions (e.g., Read for consumers, Write for producers). Use IAM conditions to restrict access to specific topics/clusters.

Code/Config Example:

{
"Effect": "Allow",
"Action": ["kafka-cluster:Read"],
"Resource": "arn:aws:kafka:us-east-1:123456789:topic/my-cluster/*/orders"
}

🔍 Interviewer Notes: Tests least privilege.

❌ Common Mistakes:

  • Wildcard permissions (*)
  • Not scoping to topics
  • Overly broad cluster permissions

Q151: How do you secure MSK ZooKeeper nodes?​

Technical Answer: MSK manages ZooKeeper security automatically. For self-managed, restrict ZooKeeper ports (2181) to broker security groups only.

Code/Config Example:

# ZooKeeper security group (MSK managed)
{
"IpProtocol": "tcp",
"FromPort": 2181,
"ToPort": 2181,
"SourceSecurityGroupId": "sg-brokers"
}

🔍 Interviewer Notes: Tests ZooKeeper security.

❌ Common Mistakes:

  • Exposing ZooKeeper to internet
  • Not restricting port access
  • Missing IAM permissions

Q152: What is MSK broker security group best practices?​

Technical Answer: Allow only required ports (9098 for TLS), restrict source to client/connector security groups, and avoid 0.0.0.0/0 inbound rules.

Code/Config Example:

# Broker security group
{
"Ingress": [{
"IpProtocol": "tcp",
"FromPort": 9098,
"ToPort": 9098,
"SourceSecurityGroupId": "sg-clients"
}]
}

🔍 Interviewer Notes: Tests security group design.

❌ Common Mistakes:

  • Allow all inbound (0.0.0.0/0)
  • Not restricting port ranges
  • Shared security groups

Q153: How do you use MSK with KMS key rotation?​

Technical Answer: Enable automatic KMS key rotation (annual) for at-rest encryption. For client-side encryption, rotate data keys regularly using KMS.

Code/Config Example:

kms.enable_key_rotation(
KeyId='arn:aws:kms:us-east-1:123456789:key/abcd1234'
)

🔍 Interviewer Notes: Tests key rotation.

❌ Common Mistakes:

  • Not enabling automatic rotation
  • Not updating IAM policies after rotation
  • Manual rotation delays

Q154: What is MSK SASL/SCRAM security?​

Technical Answer: SASL/SCRAM uses username/password stored in Secrets Manager for authentication. Credentials are encrypted in transit with TLS.

Code/Config Example:

producer = KafkaProducer(
security_protocol='SASL_SSL',
sasl_mechanism='SCRAM-SHA-512',
sasl_plain_username='user',
sasl_plain_password='${secretsmanager:msk-scram}'
)

🔍 Interviewer Notes: Tests SASL/SCRAM.

❌ Common Mistakes:

  • Not using TLS with SASL/SCRAM
  • Hardcoding credentials
  • Not rotating secrets

Q155: How do you rotate MSK SASL/SCRAM credentials?​

Technical Answer: Update credentials in Secrets Manager, update the secret version, and restart clients to fetch new credentials. No cluster downtime required.

Code/Config Example:

secretsmanager.update_secret(
SecretId='msk-scram',
SecretString=json.dumps({'username': 'user', 'password': 'new-password'})
)

🔍 Interviewer Notes: Tests credential rotation.

❌ Common Mistakes:

  • Not updating all clients
  • Hardcoding credentials
  • Not testing rotation

Q156: What is MSK IAM policy conditions?​

Technical Answer: IAM conditions restrict access by source VPC, IP address, or time of day. Use conditions to enforce additional security layers.

Code/Config Example:

{
"Effect": "Allow",
"Action": "kafka-cluster:Connect",
"Resource": "*",
"Condition": {
"IpAddress": {"aws:SourceIp": "203.0.113.0/24"}
}
}

🔍 Interviewer Notes: Tests IAM conditions.

❌ Common Mistakes:

  • Not using conditions
  • Overly broad conditions
  • Conflicting conditions

Q157: How do you use MSK with VPC Flow Logs?​

Technical Answer: Enable VPC Flow Logs to track traffic to/from MSK brokers. Use Flow Logs to detect unauthorized access or network anomalies.

Code/Config Example:

ec2.create_flow_logs(
ResourceIds=['vpc-msk'],
TrafficType='ALL',
LogDestination='arn:aws:logs:us-east-1:123456789:log-group:/aws/vpc/msk'
)

🔍 Interviewer Notes: Tests network monitoring.

❌ Common Mistakes:

  • Not enabling Flow Logs
  • Ignoring rejected traffic
  • High log storage costs

Q158: What is MSK private CA for mTLS?​

Technical Answer: Use AWS Private CA to issue client certificates for mTLS authentication. Deploy the CA certificate to brokers and client certificates to producers/consumers.

Code/Config Example:

# Issue client certificate
acm_pca.issue_certificate(
CertificateAuthorityArn='arn:aws:acm-pca:us-east-1:123456789:certificate-authority/abcd1234',
Csr='client-csr-pem',
SigningAlgorithm='SHA256WITHRSA'
)

🔍 Interviewer Notes: Tests private CA usage.

❌ Common Mistakes:

  • Not rotating CA certificates
  • Self-signed CAs without trust
  • Missing certificate revocation

Q159: How do you test MSK security posture?​

Technical Answer: Use IAM Access Analyzer to validate policies, run penetration tests (with AWS approval), and audit security group rules regularly.

Code/Config Example:

# IAM Access Analyzer
access_analyzer.create_analyzer(
analyzerName='msk-security-analyzer',
type='ACCOUNT'
)

🔍 Interviewer Notes: Tests security testing.

❌ Common Mistakes:

  • Not testing security posture
  • Ignoring IAM Access Analyzer findings
  • No penetration tests

Q160: What is MSK compliance (SOC2, HIPAA)?​

Technical Answer: MSK is SOC2, HIPAA, PCI-DSS compliant. Enable encryption, audit logging, and IAM authentication to meet compliance requirements.

Code/Config Example:

# HIPAA compliant cluster
{
"EncryptionInfo": {"EncryptionAtRest": {...}, "EncryptionInTransit": {...}},
"AuditLogging": {"Enabled": True}
}

🔍 Interviewer Notes: Tests compliance knowledge.

❌ Common Mistakes:

  • Not enabling audit logs
  • Missing encryption
  • Not documenting compliance controls

Q161: How do you optimize MSK costs?​

Technical Answer: Use right-sized instances, delete unused topics, use Serverless for spiky workloads, and leverage S3 for long-term retention instead of larger EBS volumes.

Code/Config Example:

# Delete unused topics
admin_client.delete_topics(['unused-topic'])

🔍 Interviewer Notes: Tests cost optimization.

❌ Common Mistakes:

  • Over-provisioning brokers
  • Not deleting unused topics
  • Large EBS volumes for long retention

Q162: What is MSK Serverless cost model?​

Technical Answer: MSK Serverless charges for throughput (GB/s) and storage (GB). No hourly broker fees, ideal for spiky or unpredictable workloads.

Code/Config Example:

# Serverless cost estimation
throughput_gb_per_sec = 0.1 # 100MB/s
storage_gb = 100
monthly_cost = (throughput_gb_per_sec * 3600 * 24 * 30 * 0.25) + (storage_gb * 0.10)

🔍 Interviewer Notes: Tests Serverless pricing.

❌ Common Mistakes:

  • Using Serverless for steady high throughput
  • Not monitoring throughput costs
  • Ignoring storage costs

Q163: How do you reduce MSK storage costs?​

Technical Answer: Set appropriate retention (time/size), use compacted topics for state data, and export old data to S3 for cheaper storage.

Code/Config Example:

admin_client.alter_configs([ConfigResource(
ConfigResourceType.TOPIC, 'orders',
configs={'retention.ms': '172800000'} # 2 days instead of 7
)])

🔍 Interviewer Notes: Tests storage optimization.

❌ Common Mistakes:

  • Long retention for all topics
  • Not using S3 for backups
  • Over-sized EBS volumes

Q164: How do you reduce MSK network costs?​

Technical Answer: Co-locate producers/consumers in same region/AZ as MSK, avoid cross-AZ traffic, and use compression to reduce data transfer.

Code/Config Example:

# Same AZ deployment
response = kafka.create_cluster(
BrokerNodeGroupInfo={'ClientSubnets': ['subnet-az1']} # Single AZ for lower cost
)

🔍 Interviewer Notes: Tests network cost optimization.

❌ Common Mistakes:

  • Cross-region replication costs
  • High cross-AZ traffic
  • Not using compression

Q165: What is MSK broker instance cost optimization?​

Technical Answer: Use smaller instance types for dev/test, right-size based on throughput metrics, and use Spot Instances for non-production (not supported for MSK, use Serverless instead).

Code/Config Example:

# Dev cluster with smaller instances
response = kafka.create_cluster(
BrokerNodeGroupInfo={'InstanceType': 'kafka.m5.large'} # Smaller than prod
)

🔍 Interviewer Notes: Tests instance cost optimization.

❌ Common Mistakes:

  • Production-sized dev clusters
  • Not right-sizing
  • Using expensive instances for low throughput

Q166: How do you reduce MSK Connect costs?​

Technical Answer: Use auto-scaling to match workload, use fewer MCUs for low throughput connectors, and delete unused connectors.

Code/Config Example:

kafkaconnect.update_connector(
connectorArn='arn:aws:kafkaconnect:us-east-1:123456789:connector/my-connector',
capacity={'autoScaling': {'maxWorkerCount': 2, 'mcuCount': 1}}
)

🔍 Interviewer Notes: Tests Connect cost optimization.

❌ Common Mistakes:

  • Over-provisioning workers
  • Unused connectors running
  • High MCU counts for low throughput

Q167: How do you use Reserved Instances for MSK?​

Technical Answer: MSK does not offer Reserved Instances, but you can use Savings Plans for compute. For steady workloads, Provisioned is cheaper than Serverless.

Code/Config Example:

# Savings Plan for MSK (via AWS Compute Savings Plan)
# Not directly available for MSK, use EC2 Savings Plan for underlying EC2 instances

🔍 Interviewer Notes: Tests pricing models.

❌ Common Mistakes:

  • Expecting MSK Reserved Instances
  • Using Serverless for steady high throughput
  • Not using Savings Plans

Q168: How do you monitor MSK costs?​

Technical Answer: Use AWS Cost Explorer, enable detailed billing, and tag MSK resources to track costs by project/environment.

Code/Config Example:

# Tag MSK cluster for cost tracking
kafka.update_cluster_configuration(
ClusterArn='arn:aws:kafka:us-east-1:123456789:cluster/my-cluster/abcd1234',
ConfigurationInfo={
'Tags': [{'Key': 'Environment', 'Value': 'Production'}]
}
)

🔍 Interviewer Notes: Tests cost monitoring.

❌ Common Mistakes:

  • Not tagging resources
  • Ignoring cost anomalies
  • Not using Cost Explorer

Q169: How do you reduce MSK cross-region replication costs?​

Technical Answer: Filter topics to replicate only critical data, use compression, and choose regions with lower data transfer costs.

Code/Config Example:

# MirrorMaker 2 topic filter
topics=orders,payments # Only replicate critical topics

🔍 Interviewer Notes: Tests replication cost optimization.

❌ Common Mistakes:

  • Replicating all topics
  • Not using compression
  • Cross-region data transfer waste

Q170: What is MSK cost allocation tagging?​

Technical Answer: Tag MSK clusters, connectors, and S3 buckets to allocate costs to departments/projects. Use Cost Explorer to filter by tags.

Code/Config Example:

# Tag cluster
{
"Tags": [
{"Key": "Department", "Value": "Finance"},
{"Key": "Project", "Value": "PaymentProcessing"}
]
}

🔍 Interviewer Notes: Tests cost allocation.

❌ Common Mistakes:

  • Not tagging resources
  • Inconsistent tag naming
  • Not using tags in Cost Explorer

Q171: How do you troubleshoot MSK connection issues?​

Technical Answer: Check security group rules, VPC routing, IAM permissions, and TLS certificate validity. Use telnet or nc to test broker port connectivity.

Code/Config Example:

# Test broker connectivity
telnet b-1.my-cluster.kafka.us-east-1.amazonaws.com 9098

🔍 Interviewer Notes: Tests troubleshooting basics.

❌ Common Mistakes:

  • Not checking security groups
  • Missing IAM permissions
  • Wrong broker endpoint

Q172: How do you troubleshoot high consumer lag?​

Technical Answer: Check consumer processing time, increase consumer count (up to partition count), or scale broker throughput. Monitor consumer group lag trends.

Code/Config Example:

# Check consumer processing time
import time
start = time.time()
process_message(msg)
print(f"Processing time: {time.time()-start}s")

🔍 Interviewer Notes: Tests lag troubleshooting.

❌ Common Mistakes:

  • Not scaling consumers
  • Long processing time
  • Not monitoring lag trends

Q173: How do you troubleshoot MSK producer errors?​

Technical Answer: Check acks settings, ISR count, broker health, and network connectivity. Enable producer callbacks to capture errors.

Code/Config Example:

producer.send('orders', value=order).add_errback(lambda e: print(f"Error: {e}"))

🔍 Interviewer Notes: Tests producer troubleshooting.

❌ Common Mistakes:

  • Not handling send errors
  • ISR too small for acks=all
  • Network timeouts

Q174: How do you troubleshoot MSK broker failures?​

Technical Answer: Check UnderReplicatedPartitions metric, broker CPU/memory, and CloudWatch Logs. MSK automatically replaces failed brokers, but investigate root cause.

Code/Config Example:

# Describe failed broker operation
kafka.describe_cluster_operation(ClusterOperationArn='arn:aws:kafka:...:operation/abcd1234')

🔍 Interviewer Notes: Tests broker troubleshooting.

❌ Common Mistakes:

  • Not monitoring broker health
  • Ignoring under-replicated partitions
  • Manual intervention delays

Q175: How do you troubleshoot MSK Connect failures?​

Technical Answer: Check connector logs in CloudWatch, verify IAM permissions for external systems, and test connector configuration with a small dataset.

Code/Config Example:

# Get connector logs
logs.filter_log_events(
logGroupName='/aws/kafkaconnect/my-connector',
filterPattern='ERROR'
)

🔍 Interviewer Notes: Tests Connect troubleshooting.

❌ Common Mistakes:

  • Not checking connector logs
  • Missing IAM permissions
  • Wrong connector configuration

Q176: How do you troubleshoot MSK Lambda integration?​

Technical Answer: Check Lambda function errors, event source mapping status, and MSK authentication. Verify the function can connect to MSK brokers.

Code/Config Example:

# Check event source mapping status
lambda_client.get_event_source_mapping(UUID='abcd1234')

🔍 Interviewer Notes: Tests Lambda troubleshooting.

❌ Common Mistakes:

  • Not checking function errors
  • Missing IAM permissions
  • Wrong MSK auth type

Q177: How do you troubleshoot MSK data loss?​

Technical Answer: Check topic retention settings, consumer offset commits, and ISR status. Restore from S3 backups or DR clusters if needed.

Code/Config Example:

# Check topic retention
configs = admin_client.describe_configs([ConfigResource(ConfigResourceType.TOPIC, 'orders')])

🔍 Interviewer Notes: Tests data loss troubleshooting.

❌ Common Mistakes:

  • Retention too short
  • Not committing offsets
  • No backup strategy

Q178: How do you troubleshoot MSK performance issues?​

Technical Answer: Check broker CPU/memory, network throughput, partition count, and consumer/producer configuration. Use CloudWatch metrics to identify bottlenecks.

Code/Config Example:

# Check partition count
partitions = consumer.partitions_for_topic('orders')
print(f"Partition count: {len(partitions)}")

🔍 Interviewer Notes: Tests performance troubleshooting.

❌ Common Mistakes:

  • Too few partitions
  • Broker resource exhaustion
  • Wrong producer/consumer config

Q179: How do you troubleshoot MSK rebalancing issues?​

Technical Answer: Check consumer heartbeat settings, processing time, and use CooperativeStickyAssignor to reduce rebalancing impact.

Code/Config Example:

consumer = KafkaConsumer(
partition_assignment_strategy='org.apache.kafka.clients.consumer.CooperativeStickyAssignor',
session_timeout_ms=30000, # Increase from default 10000
max_poll_interval_ms=300000
)

🔍 Interviewer Notes: Tests rebalancing troubleshooting.

❌ Common Mistakes:

  • Long processing causing session timeout
  • Not using cooperative rebalancing
  • Low session timeout

Q180: How do you troubleshoot MSK encryption issues?​

Technical Answer: Check TLS certificate validity, KMS key permissions, and client security protocol. Use openssl to test broker TLS handshake.

Code/Config Example:

# Test TLS handshake
openssl s_client -connect b-1.my-cluster.kafka.us-east-1.amazonaws.com:9098 -CAfile AmazonRootCA1.pem

🔍 Interviewer Notes: Tests encryption troubleshooting.

❌ Common Mistakes:

  • Expired certificates
  • Missing KMS permissions
  • Wrong security protocol

Q181: How do you troubleshoot MSK cluster creation failures?​

Technical Answer: Check CloudWatch Logs for cluster creation errors, verify VPC/subnet capacity, and ensure IAM role has correct permissions. Common issues: subnet full, insufficient permissions.

Code/Config Example:

# Describe failed cluster operation
kafka.describe_cluster_operation(ClusterOperationArn='arn:aws:kafka:...:operation/abcd1234')

🔍 Interviewer Notes: Tests cluster creation troubleshooting.

❌ Common Mistakes:

  • Not checking operation logs
  • Missing IAM permissions
  • Subnets full

Q182: How do you troubleshoot MSK broker replacement failures?​

Technical Answer: Check UnderReplicatedPartitions, verify new broker joins cluster, and ensure subnets have available IPs. MSK automatically retries replacement.

Code/Config Example:

# Check broker replacement status
kafka.list_cluster_operations(ClusterArn='arn:aws:kafka:...:cluster/abcd1234')

🔍 Interviewer Notes: Tests broker replacement troubleshooting.

❌ Common Mistakes:

  • Manual intervention too early
  • Not checking subnet IP availability
  • Ignoring operation errors

Q183: How do you troubleshoot MSK authentication failures?​

Technical Answer: Verify security protocol matches cluster settings, check IAM role permissions, and validate SASL/SCRAM credentials in Secrets Manager.

Code/Config Example:

# Test IAM authentication
response = kafka.get_bootstrap_brokers(ClusterArn='arn:aws:kafka:...:cluster/abcd1234')

🔍 Interviewer Notes: Tests auth troubleshooting.

❌ Common Mistakes:

  • Wrong security protocol
  • Missing IAM permissions
  • Expired SASL credentials

Q184: How do you troubleshoot MSK authorization failures?​

Technical Answer: Check IAM policies for correct actions/resources, verify ACLs if using SASL/SCRAM, and test with minimal permissions first.

Code/Config Example:

# Test with minimal policy
{
"Effect": "Allow",
"Action": ["kafka-cluster:Connect"],
"Resource": "arn:aws:kafka:*:*:cluster/my-cluster/*"
}

🔍 Interviewer Notes: Tests authorization troubleshooting.

❌ Common Mistakes:

  • Missing topic permissions
  • Wrong resource ARNs
  • Overly restrictive policies

Q185: How do you troubleshoot MSK topic creation failures?​

Technical Answer: Check broker leader availability, verify topic name compliance, and ensure enough brokers for replication factor.

Code/Config Example:

# Check cluster broker count
response = kafka.describe_cluster(ClusterArn='arn:aws:kafka:...:cluster/abcd1234')
print(response['ClusterInfo']['NumberOfBrokerNodes'])

🔍 Interviewer Notes: Tests topic creation troubleshooting.

❌ Common Mistakes:

  • Replication factor > broker count
  • Invalid topic name
  • Broker not available

Q186: How do you troubleshoot MSK partition reassignment failures?​

Technical Answer: Check partition leader availability, verify broker IDs are valid, and ensure no rebalancing is in progress.

Code/Config Example:

# Verify reassignment plan
kafka-reassign-partitions.sh --verify --reassignment-json-file plan.json

🔍 Interviewer Notes: Tests partition troubleshooting.

❌ Common Mistakes:

  • Invalid broker IDs
  • Reassignment during rebalance
  • Broker not available

Q187: How do you troubleshoot MSK high latency?​

Technical Answer: Check producer linger_ms, consumer processing time, broker CPU/memory, and network throughput. Use CloudWatch metrics to identify bottlenecks.

Code/Config Example:

# Measure end-to-end latency
import time
start = time.time()
producer.send('orders', value=order)
print(f"Produce latency: {time.time()-start}s")

🔍 Interviewer Notes: Tests latency troubleshooting.

❌ Common Mistakes:

  • High linger_ms
  • Slow consumer processing
  • Broker resource exhaustion

Q188: How do you troubleshoot MSK data corruption?​

Technical Answer: Check producer idempotence, verify consumer deserialization, and compare checksums if available. Restore from S3 backups if needed.

Code/Config Example:

# Enable idempotent producer
producer = KafkaProducer(enable_idempotence=True)

🔍 Interviewer Notes: Tests data integrity troubleshooting.

❌ Common Mistakes:

  • Not using idempotent producers
  • Wrong deserialization
  • No backup strategy

Q189: How do you troubleshoot MSK ZooKeeper issues?​

Technical Answer: MSK manages ZooKeeper automatically. Check cluster health, and if using self-managed, verify ZooKeeper ports and quorum.

Code/Config Example:

# Check cluster health (ZooKeeper managed by MSK)
response = kafka.describe_cluster(ClusterArn='arn:aws:kafka:...:cluster/abcd1234')

🔍 Interviewer Notes: Tests ZooKeeper troubleshooting.

❌ Common Mistakes:

  • Manual ZooKeeper intervention
  • Not checking cluster health
  • Port blocking

Q190: How do you troubleshoot MSK KRaft migration issues?​

Technical Answer: Verify Kafka version supports KRaft, check controller quorum health, and follow official migration guide. Test in non-prod first.

Code/Config Example:

# Check Kafka version
kafka-broker-api-versions.sh --bootstrap-server broker:9092

🔍 Interviewer Notes: Tests KRaft troubleshooting.

❌ Common Mistakes:

  • Unsupported Kafka version
  • Not testing migration
  • Quorum not healthy

Q191: How do you use MSK with Kafka Streams?​

Technical Answer: Deploy Kafka Streams applications as standard Java applications, configure bootstrap servers, and use IAM authentication if using MSK.

Code/Config Example:

Properties props = new Properties();
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "b-1.my-cluster.kafka.us-east-1.amazonaws.com:9098");
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "my-streams-app");

🔍 Interviewer Notes: Tests Kafka Streams integration.

❌ Common Mistakes:

  • Wrong bootstrap servers
  • Not configuring IAM
  • Missing application.id

Q192: What is Kafka Streams KTable?​

Technical Answer: KTable is a changelog stream representing the latest value for each key. It supports stateful operations like aggregations and joins.

Code/Config Example:

KTable<String, Long> wordCounts = source
.flatMapValues(value -> Arrays.asList(value.split("\\W+")))
.groupBy((key, value) -> value)
.count();

🔍 Interviewer Notes: Tests KTable understanding.

❌ Common Mistakes:

  • Confusing KStream and KTable
  • Not understanding changelog
  • Wrong serialization

Q193: How do you use MSK with ksqlDB?​

Technical Answer: Deploy ksqlDB as a separate cluster, connect to MSK as a datasource, and use SQL to process streams. ksqlDB is not managed by MSK.

Code/Config Example:

CREATE STREAM orders (
order_id VARCHAR,
amount DOUBLE
) WITH (
KAFKA_TOPIC='orders',
VALUE_FORMAT='JSON'
);

🔍 Interviewer Notes: Tests ksqlDB integration.

❌ Common Mistakes:

  • Not deploying ksqlDB cluster
  • Wrong topic format
  • Missing IAM permissions

Q194: What is Kafka Streams exactly-once processing?​

Technical Answer: Enable exactly-once semantics with processing.guarantee=exactly_once_v2. Requires idempotent producers and transactions.

Code/Config Example:

props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE_V2);

🔍 Interviewer Notes: Tests EOS understanding.

❌ Common Mistakes:

  • Not enabling idempotence
  • Wrong processing guarantee
  • Not handling transactions

Q195: How do you scale Kafka Streams applications?​

Technical Answer: Increase application instances up to partition count. Each instance processes a subset of partitions, so more partitions = more parallelism.

Code/Config Example:

// Scale instances to match partition count (6 partitions = 6 instances max)

🔍 Interviewer Notes: Tests scaling Streams apps.

❌ Common Mistakes:

  • More instances than partitions
  • Not increasing partitions
  • Uneven workload

Q196: What is Kafka Streams state store?​

Technical Answer: State stores persist state for stateful operations (aggregations, joins). They are backed by local RocksDB and changelog topics.

Code/Config Example:

Materialized.as("word-count-store") // Named state store

🔍 Interviewer Notes: Tests state store understanding.

❌ Common Mistakes:

  • Not naming state stores
  • Changelog topic retention too short
  • RocksDB memory issues

Q197: How do you handle Kafka Streams rebalancing?​

Technical Answer: Use cooperative rebalancing, handle rebalance listeners, and commit offsets properly. Avoid long processing during rebalances.

Code/Config Example:

props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, CooperativeStickyAssignor.class.getName());

🔍 Interviewer Notes: Tests Streams rebalancing.

❌ Common Mistakes:

  • Not using cooperative rebalancing
  • Long processing during rebalance
  • Not committing offsets

Q198: What is Kafka Streams windowing?​

Technical Answer: Windowing groups records by time for aggregations (tumbling, hopping, sliding windows). Used for time-based analytics.

Code/Config Example:

.groupByKey()
.windowedBy(TimeWindows.of(Duration.ofMinutes(5)))
.count();

🔍 Interviewer Notes: Tests windowing concepts.

❌ Common Mistakes:

  • Wrong window type
  • Not handling window retention
  • Late events not handled

Q199: How do you monitor Kafka Streams applications?​

Technical Answer: Monitor consumer lag, state store size, and custom metrics via JMX or Prometheus. Check application logs for errors.

Code/Config Example:

// Expose JMX metrics for Prometheus
props.put(StreamsConfig.METRICS_RECORDING_LEVEL_CONFIG, "DEBUG");

🔍 Interviewer Notes: Tests Streams monitoring.

❌ Common Mistakes:

  • Not monitoring consumer lag
  • Missing state store metrics
  • No error alerts

Q200: How do you tune Kafka Streams performance?​

Technical Answer: Tune commit.interval.ms, cache.max.bytes.buffering, and state store RocksDB settings. Increase parallelism with more partitions.

Code/Config Example:

props.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 30000); // 30s commit

🔍 Interviewer Notes: Tests Streams tuning.

❌ Common Mistakes:

  • Commit interval too short
  • Cache too small
  • Not enough parallelism

Q201: How do you use MSK with Schema Registry?​

Technical Answer: Deploy Confluent Schema Registry in the same VPC as MSK, configure producers/consumers to use Avro/Protobuf, and manage schema versions.

Code/Config Example:

from confluent_kafka.schema_registry import SchemaRegistryClient
registry = SchemaRegistryClient({'url': 'http://schema-registry:8081'})

🔍 Interviewer Notes: Tests Schema Registry integration.

❌ Common Mistakes:

  • Not deploying Schema Registry
  • Wrong serialization
  • Breaking schema compatibility

Q202: What is Schema Registry subject naming?​

Technical Answer: Subjects are named as [topic]-key or [topic]-value. Schema versions are tracked per subject for compatibility checks.

Code/Config Example:

# Register schema for topic "orders" value
subject = "orders-value"
schema_id = registry.register_schema(subject, order_schema)

🔍 Interviewer Notes: Tests subject naming.

❌ Common Mistakes:

  • Wrong subject name
  • Not versioning schemas
  • Ignoring compatibility

Q203: How do you use MSK with multi-region Schema Registry?​

Technical Answer: Deploy Schema Registry in each region, replicate schemas between regions, or use a central Schema Registry with cross-region access.

Code/Config Example:

# Multi-region registry config
registry = SchemaRegistryClient({
'url': 'https://registry.us-east-1.example.com:8081',
'basic.auth.user.info': 'user:pass'
})

🔍 Interviewer Notes: Tests multi-region schemas.

❌ Common Mistakes:

  • Schema mismatch between regions
  • Not replicating schemas
  • High cross-region latency

Q204: How do you use MSK with Kafka Connect Schema Registry?​

Technical Answer: Configure Connect workers with Schema Registry URL, use Avro/Protobuf converters, and reference schemas in connector config.

Code/Config Example:

connector_config = {
'key.converter': 'io.confluent.connect.avro.AvroConverter',
'value.converter': 'io.confluent.connect.avro.AvroConverter',
'key.converter.schema.registry.url': 'http://schema-registry:8081'
}

🔍 Interviewer Notes: Tests Connect Schema Registry.

❌ Common Mistakes:

  • Missing converter config
  • Wrong registry URL
  • Not registering schemas

Q205: How do you use MSK with AWS Glue Schema Registry?​

Technical Answer: Use Glue Schema Registry instead of Confluent, configure producers/consumers with Glue serializers, and manage schemas via Glue.

Code/Config Example:

from aws_glue_schema_registry import SchemaRegistryClient
registry = SchemaRegistryClient({'region': 'us-east-1'})

🔍 Interviewer Notes: Tests Glue Schema Registry.

❌ Common Mistakes:

  • Not using Glue IAM permissions
  • Wrong serializer
  • Not creating schemas

Q206: How do you migrate schemas to MSK?​

Technical Answer: Export schemas from existing Registry, import to MSK-connected Registry, and update producers/consumers to use new Registry URL.

Code/Config Example:

# Export schema
schema = registry.get_latest_version(subject)
# Import to new registry
new_registry.register_schema(subject, schema.schema)

🔍 Interviewer Notes: Tests schema migration.

❌ Common Mistakes:

  • Schema compatibility breaks
  • Not testing migration
  • Wrong Registry URL

Q207: How do you use MSK with Protobuf?​

Technical Answer: Define Protobuf schemas, register with Schema Registry, and use Protobuf serializer/deserializer in producers/consumers.

Code/Config Example:

from confluent_kafka.schema_registry.protobuf import ProtobufSerializer
serializer = ProtobufSerializer(order_schema, registry)

🔍 Interviewer Notes: Tests Protobuf usage.

❌ Common Mistakes:

  • Not registering Protobuf schemas
  • Wrong serializer
  • Compatibility issues

Q208: How do you handle Schema Registry high availability?​

Technical Answer: Deploy multiple Schema Registry instances, use load balancer, and configure Kafka changelog topic for Registry state.

Code/Config Example:

# Schema Registry HA config
kafkastore.bootstrap.servers=broker:9092
host.name=schema-registry-1

🔍 Interviewer Notes: Tests Registry HA.

❌ Common Mistakes:

  • Single Registry instance
  • No load balancer
  • Changelog topic not replicated

Q209: How do you use MSK with JSON Schema?​

Technical Answer: Use JSON Schema with Schema Registry, configure producers/consumers to validate JSON against schema, and register JSON schemas.

Code/Config Example:

from confluent_kafka.schema_registry.json_schema import JSONSchemaSerializer
serializer = JSONSchemaSerializer(registry, order_schema)

🔍 Interviewer Notes: Tests JSON Schema usage.

❌ Common Mistakes:

  • Not validating JSON
  • Schema not registered
  • Wrong converter

Q210: How do you troubleshoot Schema Registry issues?​

Technical Answer: Check Registry logs, verify connectivity to MSK, validate schema compatibility, and test schema registration.

Code/Config Example:

# Test Registry connectivity
curl http://schema-registry:8081/subjects

🔍 Interviewer Notes: Tests Registry troubleshooting.

❌ Common Mistakes:

  • Not checking Registry logs
  • Connectivity issues
  • Compatibility errors

Q211: How do you use MSK multi-region replication with MirrorMaker 2?​

Technical Answer: Deploy MirrorMaker 2 in a separate cluster, configure source/destination clusters, and replicate topics with filters.

Code/Config Example:

# MirrorMaker 2 config
source.cluster.bootstrap.servers=primary:9092
target.cluster.bootstrap.servers=dr:9092

🔍 Interviewer Notes: Tests MirrorMaker 2 setup.

❌ Common Mistakes:

  • Not filtering topics
  • High cross-region costs
  • Replication lag

Q212: How do you monitor MSK multi-region replication?​

Technical Answer: Monitor replication lag, cross-region data transfer, and MirrorMaker 2 worker metrics. Set alerts for high lag.

Code/Config Example:

# Monitor MirrorMaker 2 metrics
cloudwatch.get_metric_statistics(Namespace='KafkaConnect', MetricName='RecordsLag')

🔍 Interviewer Notes: Tests replication monitoring.

❌ Common Mistakes:

  • Not monitoring lag
  • Ignoring data transfer costs
  • Worker failures

Q213: How do you use MSK with Confluent Replicator?​

Technical Answer: Deploy Confluent Replicator as a Kafka Connect connector, configure source/destination clusters, and manage replication via Confluent Control Center.

Code/Config Example:

connector_config = {
'connector.class': 'io.confluent.connect.replicator.ReplicatorSourceConnector',
'src.kafka.bootstrap.servers': 'primary:9092',
'dest.kafka.bootstrap.servers': 'dr:9092'
}

🔍 Interviewer Notes: Tests Confluent Replicator.

❌ Common Mistakes:

  • Not using Confluent connector
  • Wrong bootstrap servers
  • Missing IAM permissions

Q214: How do you handle MSK cross-region failover?​

Technical Answer: Use Route53 failover records, update client bootstrap servers, or use MSK Serverless with multi-region endpoints.

Code/Config Example:

# Failover clients to DR cluster
bootstrap_servers = ['b-1.dr.kafka.us-west-2.amazonaws.com:9098']

🔍 Interviewer Notes: Tests cross-region failover.

❌ Common Mistakes:

  • Manual failover delay
  • Not testing failover
  • DNS caching

Q215: How do you use MSK with Global Accelerator?​

Technical Answer: Deploy MSK behind Global Accelerator for low-latency global access. Accelerator routes traffic to the nearest MSK region.

Code/Config Example:

# Global Accelerator endpoint
bootstrap_servers = ['ga-1234.amazonaws.com:9098']

🔍 Interviewer Notes: Tests Global Accelerator integration.

❌ Common Mistakes:

  • Not configuring endpoints
  • High accelerator costs
  • TCP vs HTTP accelerators

Technical Answer: Create VPC endpoints in client VPC pointing to MSK in another region via PrivateLink. Requires VPC endpoint service in MSK region.

Code/Config Example:

# Cross-region PrivateLink endpoint
ec2.create_vpc_endpoint(ServiceName='com.amazonaws.us-west-2.kafka-cluster')

🔍 Interviewer Notes: Tests cross-region PrivateLink.

❌ Common Mistakes:

  • Not creating endpoint service
  • Wrong region service name
  • Security group issues

Q217: How do you comply with MSK data residency requirements?​

Technical Answer: Deploy MSK in required region, enable encryption at rest, use KMS CMK, and restrict cross-region replication for sensitive data.

Code/Config Example:

# Region-specific cluster
response = kafka.create_cluster(Region='eu-central-1') # EU data residency

🔍 Interviewer Notes: Tests data residency compliance.

❌ Common Mistakes:

  • Cross-region replication of sensitive data
  • Not using KMS CMK
  • Missing compliance audits

Q218: How do you use MSK with AWS Audit Manager?​

Technical Answer: Enable MSK audit logging, map logs to Audit Manager controls, and generate compliance reports for SOC2, HIPAA.

Code/Config Example:

# Enable audit logging for compliance
kafka.update_cluster_configuration(ConfigurationInfo={'AuditLogging': {'Enabled': True}})

🔍 Interviewer Notes: Tests Audit Manager integration.

❌ Common Mistakes:

  • Not enabling audit logs
  • Not mapping controls
  • Missing reports

Q219: How do you migrate from self-managed Kafka to MSK?​

Technical Answer: Use MirrorMaker 2 to replicate topics, switch clients to MSK bootstrap servers, and decommission self-managed cluster after validation.

Code/Config Example:

# Migrate with MirrorMaker 2
kafka-mirror-maker.sh --source.properties source.properties --target.properties target.properties

🔍 Interviewer Notes: Tests migration planning.

❌ Common Mistakes:

  • Not testing migration
  • Downtime during cutover
  • Data loss

Q220: How do you upgrade MSK Kafka version?​

Technical Answer: Use UpdateClusterKafkaVersion API, perform rolling upgrade, and test client compatibility with new version.

Code/Config Example:

kafka.update_cluster_kafka_version(
ClusterArn='arn:aws:kafka:...:cluster/abcd1234',
TargetKafkaVersion='3.6.0'
)

🔍 Interviewer Notes: Tests version upgrades.

❌ Common Mistakes:

  • Not testing compatibility
  • Rolling upgrade issues
  • Downtime

3. Advanced Topics​

Q221: How do you implement multi-region MSK replication?​

Technical Answer: Use MirrorMaker 2 or custom replication.

MirrorMaker 2:

# Run MirrorMaker 2 to replicate between clusters
./bin/kafka-mirror-maker.sh \
--consumer.config source.properties \
--producer.config target.properties \
--whitelist "orders.*"

🔍 Interviewer Notes: Testing disaster recovery patterns.


Q222: How do you tune MSK for high throughput?​

Technical Answer:

Producer Tuning:

producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
batch_size=32768, # Increase batch size
linger_ms=100, # Wait to batch more
compression_type='lz4', # Enable compression
buffer_memory=67108864, # 64MB buffer
max_in_flight_requests_per_connection=5,
acks='1' # Or 'all' for durability
)

Consumer Tuning:

consumer = KafkaConsumer(
'orders',
bootstrap_servers=bootstrap_servers,
max_poll_records=1000,
max_poll_interval_ms=300000,
fetch_min_bytes=100000,
fetch_max_wait_ms=500
)

🔍 Interviewer Notes: Testing performance optimization.


Section 4: Extended Advanced Questions​

Q223: How do you implement exactly-once semantics (EOS) in Kafka?​

Technical Answer: Kafka supports exactly-once semantics through idempotent producers and transactions.

Idempotent Producer:

from kafka import KafkaProducer

producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM',
sasl_aws_msk_iam_auth_token_provider=get_auth_token,

# Exactly-once settings
enable_idempotence=True, # Enable idempotent producer
acks='all', # Wait for all ISRs
retries=5, # Retry on transient failures
max_in_flight_requests_per_connection=5, # Must be <=5 for idempotence

# Transactional settings
transactional_id='my-producer-1' # Unique per producer instance
)

# Init transactions
producer.init_transactions()

try:
producer.begin_transaction()

# Send messages
for record in records:
producer.send('output-topic', record)

# Commit transaction
producer.commit_transaction()
except Exception as e:
producer.abort_transaction()
raise

Kafka Streams Exactly-Once:

# Kafka Streams EOS configuration
props = {
'bootstrap.servers': '...',
'application.id': 'my-streams-app',
'processing.guarantee': 'exactly_once_v2', # Enable EOS
'isolation.level': 'read_committed', # Only read committed messages
}

🔍 Interviewer Notes: Testing understanding of distributed transaction consistency.

❌ Common Mistakes:

  • Not enabling idempotence
  • Using retries > max_in_flight without idempotence
  • Not handling transaction timeouts

Q224: What is Kafka Schema Registry and how do you use it?​

Technical Answer: Schema Registry manages Avro/JSON/Protobuf schemas for Kafka messages.

from confluent_kafka import Producer, Consumer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer, AvroDeserializer

# Schema Registry client
schema_registry_conf = {'url': 'http://schema-registry:8081'}
schema_registry_client = SchemaRegistryClient(schema_registry_conf)

# Define schema
order_schema = '''
{
"type": "record",
"name": "Order",
"fields": [
{"name": "order_id", "type": "string"},
{"name": "customer_id", "type": "string"},
{"name": "amount", "type": "double"}
]
}
'''

# Producer with schema
avro_serializer = AvroSerializer(
schema_registry_client,
order_schema
)

producer = Producer({
'bootstrap.servers': bootstrap_servers,
'key.serializer': None,
'value.serializer': avro_serializer
})

# Send message (serialized with schema)
producer.produce(
'orders',
key='order-123',
value={'order_id': '123', 'customer_id': '456', 'amount': 99.99}
)

Schema Evolution:

  • BACKWARD: New schema can read old data
  • FORWARD: Old schema can read new data
  • FULL: Both backward and forward
  • NONE: No compatibility checking

🔍 Interviewer Notes: Testing data governance and schema management.

❌ Common Mistakes:

  • Not using Schema Registry (manual serialization)
  • Breaking schema compatibility
  • Not versioning schemas

Q225: What is KRaft mode and how does it differ from ZooKeeper?​

Technical Answer: KRaft (Kafka Raft) removes ZooKeeper dependency, using internal Raft consensus.

KRaft Architecture:

┌──────────────────────────────────────────────┐
│ Kafka Cluster (KRaft) │
├──────────────────────────────────────────────┤
│ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │Controller│ │Controller│ │Controller│ │
│ │ (Quorum)│ │ (Quorum)│ │ (Quorum)│ │
│ └─────────┘ └─────────┘ └─────────┘ │
│ │ │ │ │
│ └───────────┴───────────┘ │
│ Metadata Quorum │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Broker │ │ Broker │ │ Broker │ │
│ │ 1 │ │ 2 │ │ 3 │ │
│ └─────────┘ └─────────┘ └─────────┘ │
└──────────────────────────────────────────────┘

Benefits:

  • Simplified deployment (no ZooKeeper)
  • Better scalability (millions of partitions)
  • Faster recovery (seconds vs minutes)
  • Single security model

Migration from ZooKeeper:

# KRaft migration tool
kafka-storage.sh format \
--config server.properties \
--cluster-id $(kafka-storage.sh random-uuid)

# Start in KRaft mode
kafka-server-start.sh server.properties

🔍 Interviewer Notes: Testing knowledge of modern Kafka architecture.

❌ Common Mistakes:

  • Mixing ZooKeeper and KRaft concepts
  • Not understanding quorum controllers
  • Wrong cluster ID during migration

Q226: How do you implement advanced Kafka security patterns?​

Technical Answer: Kafka security uses SASL for authentication and ACLs for authorization.

IAM Authentication (MSK):

from kafka import KafkaProducer
from aws_msk_iam_sasl_signer import MSKAuthTokenProvider

def get_auth_token():
return MSKAuthTokenProvider.generate_auth_token('us-east-1')

producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL',
sasl_mechanism='AWS_MSK_IAM',
sasl_aws_msk_iam_auth_token_provider=get_auth_token
)

SCRAM Authentication:

producer = KafkaProducer(
bootstrap_servers=bootstrap_servers,
security_protocol='SASL_SSL',
sasl_mechanism='SCRAM-SHA-512',
sasl_plain_username='user',
sasl_plain_password='password'
)

ACLs (Authorization):

# Grant permissions
kafka-acls.sh --bootstrap-server broker:9092 \
--add \
--allow-principal User:alice \
--operation Read \
--operation Write \
--topic orders

# List ACLs
kafka-acls.sh --bootstrap-server broker:9092 \
--list \
--topic orders

🔍 Interviewer Notes: Testing security configuration and best practices.

❌ Common Mistakes:

  • Not enabling authentication on MSK
  • Using PLAINTEXT in production
  • Overly permissive ACLs

Q227: How do you optimize Kafka log compaction for production?​

Technical Answer: Log compaction retains the latest value for each key, useful for changelog topics.

Enabling Compaction:

# Create compacted topic
kafka-topics.sh --create \
--bootstrap-server localhost:9092 \
--topic user-preferences \
--partitions 3 \
--replication-factor 3 \
--config cleanup.policy=compact \
--config min.cleanable.dirty.ratio=0.5 \
--config segment.ms=100 \
--config delete.retention.ms=100

Python with Compacted Topic:

# Produce to compacted topic (key is required)
producer.send(
'user-preferences',
key='user-123'.encode(), # Key for compaction
value=json.dumps({'theme': 'dark', 'lang': 'en'})
)

# Consumer reads latest value per key
consumer = KafkaConsumer(
'user-preferences',
bootstrap_servers=bootstrap_servers,
auto_offset_reset='earliest', # Read from beginning
key_deserializer=lambda k: k.decode('utf-8'),
value_deserializer=lambda v: json.loads(v)
)

for msg in consumer:
print(f"User: {msg.key}, Preferences: {msg.value}")

Use Cases:

  • Database changelog (CDC)
  • Configuration updates
  • State stores for Kafka Streams

🔍 Interviewer Notes: Testing understanding of retention policies.

❌ Common Mistakes:

  • Not setting keys (required for compaction)
  • Mixing compact and delete policies incorrectly
  • Not understanding min.cleanable.dirty.ratio

Q228: How do you version schemas in MSK?​

Technical Answer: Schemas are versioned per subject (topic-key/topic-value) in Schema Registry. New versions are registered with compatibility checks.

Code/Config Example:

# Register new schema version
new_schema = '{"type": "record", "name": "Order", "fields": [...]}'
schema_id = registry.register_schema("orders-value", new_schema)

🔍 Interviewer Notes: Tests schema versioning.

❌ Common Mistakes:

  • Not versioning schemas
  • Breaking compatibility
  • Wrong subject name

Q229: How do you evolve Avro schemas in MSK?​

Technical Answer: Add optional fields with defaults, remove fields only if not used. Follow backward/forward compatibility rules.

Code/Config Example:

# Add new field with default
new_schema = """
{
"type": "record",
"name": "Order",
"fields": [
{"name": "order_id", "type": "string"},
{"name": "new_field", "type": "string", "default": ""}
]
}
"""

🔍 Interviewer Notes: Tests Avro evolution.

❌ Common Mistakes:

  • Removing fields without default
  • Changing field types
  • No default for new fields

Q230: What is Schema Registry backward compatibility?​

Technical Answer: Backward compatibility allows new schema to read old data. Old consumers can read data written by new producers.

Code/Config Example:

registry.update_compatibility("orders-value", "BACKWARD")

🔍 Interviewer Notes: Tests compatibility modes.

❌ Common Mistakes:

  • Using wrong compatibility
  • Not testing evolution
  • Breaking changes

Q231: What is Schema Registry forward compatibility?​

Technical Answer: Forward compatibility allows old schema to read new data. New consumers can read data written by old producers.

Code/Config Example:

registry.update_compatibility("orders-value", "FORWARD")

🔍 Interviewer Notes: Tests forward compatibility.

❌ Common Mistakes:

  • Confusing forward/backward
  • Not testing
  • Breaking changes

Q232: What is Schema Registry full compatibility?​

Technical Answer: Full compatibility is both backward and forward. Requires no breaking changes in either direction.

Code/Config Example:

registry.update_compatibility("orders-value", "FULL")

🔍 Interviewer Notes: Tests full compatibility.

❌ Common Mistakes:

  • Strictest mode
  • Not testing
  • Overly restrictive

Q233: How do you delete schemas in MSK?​

Technical Answer: Delete specific schema versions or the entire subject. Deleted schemas cannot be recovered.

Code/Config Example:

# Delete specific version
registry.delete_schema_version("orders-value", version=5)

# Delete entire subject
registry.delete_subject("orders-value")

🔍 Interviewer Notes: Tests schema deletion.

❌ Common Mistakes:

  • Accidental deletion
  • Not backing up schemas
  • Cannot recover

Q234: How do you use Schema Registry with Protobuf?​

Technical Answer: Define Protobuf messages, register with Schema Registry, and use Protobuf serializer/deserializer.

Code/Config Example:

from confluent_kafka.schema_registry.protobuf import ProtobufSerializer
serializer = ProtobufSerializer(order_proto, registry)

🔍 Interviewer Notes: Tests Protobuf usage.

❌ Common Mistakes:

  • Not registering Protobuf schemas
  • Wrong serializer
  • Compatibility issues

Q235: How do you use Schema Registry with JSON Schema?​

Technical Answer: Define JSON Schema, register with Registry, and use JSON Schema serializer/deserializer.

Code/Config Example:

from confluent_kafka.schema_registry.json_schema import JSONSchemaSerializer
serializer = JSONSchemaSerializer(registry, order_json_schema)

🔍 Interviewer Notes: Tests JSON Schema usage.

❌ Common Mistakes:

  • Not validating JSON
  • Schema not registered
  • Wrong converter

Q236: How do you troubleshoot Schema Registry MSK issues?​

Technical Answer: Check Registry logs, verify connectivity to MSK, validate schema compatibility, and test registration.

Code/Config Example:

curl http://schema-registry:8081/subjects

🔍 Interviewer Notes: Tests Registry troubleshooting.

❌ Common Mistakes:

  • Not checking logs
  • Connectivity issues
  • Compatibility errors

Q237: How do you secure Schema Registry?​

Technical Answer: Use TLS for Registry REST API, IAM roles for AWS Glue Registry, and restrict access via security groups.

Code/Config Example:

# Registry TLS config
listeners: https://0.0.0.0:8081
ssl.truststore.location: truststore.jks

🔍 Interviewer Notes: Tests Registry security.

❌ Common Mistakes:

  • Unencrypted Registry API
  • Open access
  • No auth

Q238: How do you monitor Schema Registry?​

Technical Answer: Monitor Registry REST API latency, schema count, and compatibility errors. Use JMX metrics or CloudWatch for Glue Registry.

Code/Config Example:

# Glue Registry metrics
cloudwatch.get_metric_statistics(Namespace='AWS/Glue')

🔍 Interviewer Notes: Tests Registry monitoring.

❌ Common Mistakes:

  • Not monitoring
  • Missing error alerts
  • Schema growth

Q239: How do you use Schema Registry with Kafka Connect?​

Technical Answer: Configure Connect workers with Registry URL, use Avro/Protobuf converters, and reference schemas in connector config.

Code/Config Example:

connector_config = {
'key.converter': 'io.confluent.connect.avro.AvroConverter',
'key.converter.schema.registry.url': 'http://registry:8081'
}

🔍 Interviewer Notes: Tests Connect Registry integration.

❌ Common Mistakes:

  • Missing converter config
  • Wrong URL
  • Not registering schemas

Q240: How do you handle Schema Registry downtime?​

Technical Answer: Deploy Registry in HA mode, cache schemas on clients, and use default values for missing schemas.

Code/Config Example:

# Cache schema locally
schema = registry.get_latest_version(subject)
cache[subject] = schema

🔍 Interviewer Notes: Tests Registry HA.

❌ Common Mistakes:

  • Single Registry instance
  • No caching
  • Downtime impact

Q241: How do you set up MSK cross-region replication?​

Technical Answer: Deploy MirrorMaker 2 in a separate cluster, configure source/destination MSK clusters, and replicate critical topics.

Code/Config Example:

# MirrorMaker 2 source config
clusters=primary.bootstrap.servers=primary:9092
clusters=dr.bootstrap.servers=dr:9092
primary->dr.enabled=true

🔍 Interviewer Notes: Tests cross-region setup.

❌ Common Mistakes:

  • Replicating all topics
  • High costs
  • Lag

Q242: How do you monitor MSK cross-region replication lag?​

Technical Answer: Monitor MirrorMaker 2 worker metrics (records-lag), check replica consumer lag, and alert on high lag.

Code/Config Example:

cloudwatch.get_metric_statistics(
Namespace='KafkaConnect',
MetricName='RecordsLag'
)

🔍 Interviewer Notes: Tests replication monitoring.

❌ Common Mistakes:

  • Not monitoring lag
  • High lag ignored
  • No alerts

Q243: How do you handle MSK cross-region failover?​

Technical Answer: Use Route53 failover, switch clients to DR cluster, and validate data consistency after failover.

Code/Config Example:

# Failover to DR
bootstrap_servers = ['dr-cluster:9098']

🔍 Interviewer Notes: Tests failover procedures.

❌ Common Mistakes:

  • Manual failover delay
  • Not testing
  • Data loss

Q244: How do you use MSK with AWS RAM?​

Technical Answer: Share MSK clusters across accounts using Resource Access Manager, grant cross-account IAM permissions.

Code/Config Example:

ram.create_resource_share(
Name='msk-share',
ResourceArns=['arn:aws:kafka:...:cluster/my-cluster']
)

🔍 Interviewer Notes: Tests RAM usage.

❌ Common Mistakes:

  • Not sharing
  • Wrong IAM permissions
  • Cross-account access issues

Technical Answer: Create VPC endpoint service in MSK account, share with client account, and create endpoint in client VPC.

Code/Config Example:

# Client account endpoint
ec2.create_vpc_endpoint(ServiceName='com.amazonaws.us-east-1.kafka-cluster')

🔍 Interviewer Notes: Tests PrivateLink cross-account.

❌ Common Mistakes:

  • Not sharing endpoint service
  • Wrong service name
  • Security group issues

Q246: How do you comply with MSK data residency?​

Technical Answer: Deploy MSK in required region, enable encryption, restrict cross-region replication, and audit data movement.

Code/Config Example:

# Region-specific cluster
kafka.create_cluster(Region='eu-central-1')

🔍 Interviewer Notes: Tests data residency.

❌ Common Mistakes:

  • Cross-region replication
  • Not encrypting
  • No audits

Q247: How do you enable MSK audit logging?​

Technical Answer: Enable audit logging via MSK configuration, logs are sent to CloudWatch Logs for compliance tracking.

Code/Config Example:

kafka.update_cluster_configuration(
ConfigurationInfo={'AuditLogging': {'Enabled': True}}
)

🔍 Interviewer Notes: Tests audit logging.

❌ Common Mistakes:

  • Not enabling
  • Not monitoring logs
  • High costs

Q248: How do you use MSK with CloudTrail?​

Technical Answer: MSK API calls are logged to CloudTrail automatically. Use CloudTrail to audit cluster operations.

Code/Config Example:

cloudtrail.lookup_events(LookupAttributes=[{'Key': 'EventName', 'Value': 'CreateCluster'}])

🔍 Interviewer Notes: Tests CloudTrail integration.

❌ Common Mistakes:

  • Not enabling CloudTrail
  • Not monitoring
  • Missing logs

Q249: How do you migrate from self-managed Kafka to MSK?​

Technical Answer: Use MirrorMaker 2 to replicate topics, test client compatibility, switch producers/consumers to MSK, decommission old cluster.

Code/Config Example:

# Migrate with MirrorMaker 2
kafka-mirror-maker.sh --source.properties source.properties --target.properties target.properties

🔍 Interviewer Notes: Tests migration planning.

❌ Common Mistakes:

  • Not testing
  • Downtime
  • Data loss

Q250: How do you upgrade MSK Kafka version?​

Technical Answer: Use UpdateClusterKafkaVersion API for rolling upgrade, test client compatibility with new version first.

Code/Config Example:

kafka.update_cluster_kafka_version(
ClusterArn='arn:aws:kafka:...:cluster/my-cluster/abcd1234',
TargetKafkaVersion='3.6.0'
)

🔍 Interviewer Notes: Tests version upgrades.

❌ Common Mistakes:

  • Not testing
  • Compatibility issues
  • Downtime

Q251: How do you handle MSK broker maintenance?​

Technical Answer: MSK performs automated maintenance, but you can schedule maintenance windows and monitor maintenance events.

Code/Config Example:

kafka.update_broker_type(
ClusterArn='arn:aws:kafka:...:cluster/my-cluster/abcd1234',
CurrentVersion='ABC123',
BrokerNodeGroupInfo={'InstanceType': 'kafka.m5.xlarge'}
)

🔍 Interviewer Notes: Tests maintenance handling.

❌ Common Mistakes:

  • Not monitoring maintenance
  • Scheduling conflicts
  • Not testing

Q252: How do you use MSK with Kafka Streams interactive queries?​

Technical Answer: Expose Kafka Streams state stores via REST API, query state from other applications.

Code/Config Example:

KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();
ReadOnlyKeyValueStore<String, Long> store = streams.store(
StoreQueryParameters.fromNameAndType("word-count-store", QueryableStoreTypes.keyValueStore())
);

🔍 Interviewer Notes: Tests interactive queries.

❌ Common Mistakes:

  • Not exposing store
  • Scaling issues
  • State store not queryable

Q253: How do you handle Kafka Streams state store recovery?​

Technical Answer: State stores recover from changelog topics on restart. Ensure changelog topic has enough replication and retention.

Code/Config Example:

// Changelog topic config
Materialized.as("store").withLoggingEnabled(configs);

🔍 Interviewer Notes: Tests state recovery.

❌ Common Mistakes:

  • Changelog topic too short
  • Not replicated
  • Recovery time

Q254: How do you use MSK with ksqlDB?​

Technical Answer: Deploy ksqlDB server, connect to MSK as datasource, run SQL queries for stream processing.

Code/Config Example:

CREATE STREAM orders (order_id VARCHAR, amount DOUBLE) WITH (KAFKA_TOPIC='orders', VALUE_FORMAT='JSON');

🔍 Interviewer Notes: Tests ksqlDB usage.

❌ Common Mistakes:

  • Not deploying ksqlDB
  • Wrong topic format
  • Missing IAM

Q255: How do you scale ksqlDB with MSK?​

Technical Answer: Add more ksqlDB nodes, balance persistent queries across nodes, and monitor query performance.

Code/Config Example:

# ksqlDB scaling
ksql.service.liquidity.min.tasks.max=16

🔍 Interviewer Notes: Tests ksqlDB scaling.

❌ Common Mistakes:

  • Under-provisioning
  • Not balancing queries
  • High latency

Q256: How do you handle MSK multi-cluster deployments?​

Technical Answer: Deploy separate clusters per environment/team, use MirrorMaker 2 for data sharing, and centralize Schema Registry.

Code/Config Example:

# Multi-cluster setup
clusters = ['cluster-dev', 'cluster-prod']

🔍 Interviewer Notes: Tests multi-cluster patterns.

❌ Common Mistakes:

  • Shared clusters
  • No data isolation
  • Complex routing

Q257: How do you use MSK with event sourcing?​

Technical Answer: Store all state changes as events in MSK topics, use compacted topics for latest state, and replay events for recovery.

Code/Config Example:

# Event sourcing topic
admin_client.create_topics([NewTopic('user-events', configs={'cleanup.policy': 'compact'})])

🔍 Interviewer Notes: Tests event sourcing.

❌ Common Mistakes:

  • Not compacting topics
  • Event versioning
  • Replay issues

Q258: How do you use MSK with CQRS?​

Technical Answer: Use MSK to separate command and query sides, process commands to update state, and publish events for query models.

Code/Config Example:

# Command topic
producer.send('commands', key=user_id, value=command)
# Query topic
consumer = KafkaConsumer('query-model')

🔍 Interviewer Notes: Tests CQRS patterns.

❌ Common Mistakes:

  • Not separating topics
  • Eventual consistency
  • Complexity

Q259: How do you test MSK applications?​

Technical Answer: Use embedded Kafka for unit tests, test containers for integration tests, and chaos testing for reliability.

Code/Config Example:

// Embedded Kafka for tests
EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(1);

🔍 Interviewer Notes: Tests testing strategies.

❌ Common Mistakes:

  • Not testing
  • No chaos testing
  • Integration test gaps

Q260: How do you use MSK with Spring Boot?​

Technical Answer: Use Spring Kafka to auto-configure producers/consumers, manage lifecycle, and handle errors.

Code/Config Example:

@Autowired
KafkaTemplate<String, Order> kafkaTemplate;

kafkaTemplate.send("orders", order);

🔍 Interviewer Notes: Tests Spring Boot integration.

❌ Common Mistakes:

  • Not auto-configuring
  • Lifecycle issues
  • Error handling

Q261: How do you use MSK with AWS CloudFormation?​

Technical Answer: Define MSK clusters, connectors, and event source mappings as CloudFormation resources for infrastructure as code.

Code/Config Example:

# CloudFormation MSK resource
MyMSKCluster:
Type: AWS::Kafka::Cluster
Properties:
ClusterName: my-cluster
KafkaVersion: 3.5.1

🔍 Interviewer Notes: Tests CloudFormation usage.

❌ Common Mistakes:

  • Not using IaC
  • Manual changes
  • Drift

Q262: How do you use MSK with Terraform?​

Technical Answer: Define MSK resources using Terraform AWS provider, manage state, and automate deployments.

Code/Config Example:

resource "aws_msk_cluster" "example" {
cluster_name = "my-cluster"
kafka_version = "3.5.1"
}

🔍 Interviewer Notes: Tests Terraform integration.

❌ Common Mistakes:

  • Not using IaC
  • State issues
  • Manual changes

Q263: How do you handle MSK client connection pooling?​

Technical Answer: Reuse producer/consumer instances across threads, use connection pools for databases accessed from consumers.

Code/Config Example:

# Singleton producer
producer = KafkaProducer(...)

🔍 Interviewer Notes: Tests connection pooling.

❌ Common Mistakes:

  • Creating new producers per message
  • Not reusing instances
  • Resource leak

Q264: How do you handle MSK large messages?​

Technical Answer: Increase broker/topic max.message.bytes, use compression, or split large messages into smaller chunks.

Code/Config Example:

admin_client.alter_configs([ConfigResource(
ConfigResourceType.TOPIC, 'orders',
configs={'max.message.bytes': '10485760'} # 10MB
)])

🔍 Interviewer Notes: Tests large message handling.

❌ Common Mistakes:

  • Default 1MB limit
  • Not compressing
  • Splitting issues

Technical Answer: Use Flink Kafka connector to read/write MSK topics, configure Flink job with MSK bootstrap servers.

Code/Config Example:

FlinkKafkaConsumer<String> consumer = new FlinkKafkaConsumer<>(
"orders",
new SimpleStringSchema(),
props
);

🔍 Interviewer Notes: Tests Flink integration.

❌ Common Mistakes:

  • Wrong connector
  • Not configuring IAM
  • Checkpointing

Q266: How do you use MSK with AWS Glue?​

Technical Answer: Use Glue jobs to read/write MSK topics via Glue Kafka connector, process data with Spark.

Code/Config Example:

# Glue job with Kafka
glue_context.create_dynamic_frame.from_options(
connection_type='kafka',
connection_options={'bootstrap.servers': 'broker:9098', 'topic': 'orders'}
)

🔍 Interviewer Notes: Tests Glue integration.

❌ Common Mistakes:

  • Not using Glue connector
  • IAM permissions
  • Schema issues

Q267: How do you handle MSK topic deletion?​

Technical Answer: Enable topic deletion via delete.topic.enable=true, use admin client to delete, and clean up client references.

Code/Config Example:

admin_client.delete_topics(['unused-topic'])

🔍 Interviewer Notes: Tests topic deletion.

❌ Common Mistakes:

  • Not enabling deletion
  • Accidental deletion
  • Client errors

Q268: How do you use MSK with AWS Step Functions?​

Technical Answer: Use Step Functions to orchestrate Lambda functions that process MSK events, handle retries and error handling.

Code/Config Example:

{
"Comment": "Process MSK events via Step Functions",
"StartAt": "ProcessOrder",
"States": {
"ProcessOrder": {
"Type": "Task",
"Resource": "arn:aws:lambda:...",
"End": true
}
}
}

🔍 Interviewer Notes: Tests Step Functions integration.

❌ Common Mistakes:

  • Not orchestrating
  • Retry handling
  • Error propagation

Q269: How do you handle MSK backpressure?​

Technical Answer: Monitor consumer lag, slow down producers, increase consumers, or implement rate limiting in producers.

Code/Config Example:

# Producer rate limiting
time.sleep(0.1) # 10 messages/sec
producer.send('orders', value=order)

🔍 Interviewer Notes: Tests backpressure handling.

❌ Common Mistakes:

  • Not monitoring lag
  • Producers overwhelming consumers
  • No rate limiting

Q270: How do you use MSK with Amazon Redshift?​

Technical Answer: Use MSK Connect Redshift sink connector, or copy data from S3 (via S3 sink) to Redshift.

Code/Config Example:

connector_config = {
'connector.class': 'com.redshift.connector.RedshiftSinkConnector',
'topics': 'orders'
}

🔍 Interviewer Notes: Tests Redshift integration.

❌ Common Mistakes:

  • Not using connector
  • IAM permissions
  • Data format

Q271: How do you handle MSK idempotent consumers?​

Technical Answer: Use transactional producers, store processed offsets in external store (DB), and check for duplicate processing.

Code/Config Example:

# Check duplicate in DB
if db.check_processed(msg.offset):
continue
process(msg)
db.mark_processed(msg.offset)

🔍 Interviewer Notes: Tests idempotent processing.

❌ Common Mistakes:

  • Not handling duplicates
  • External store issues
  • Performance impact

Q272: How do you use MSK with AWS Lambda destinations?​

Technical Answer: Configure Lambda event source mapping with destinations for success/failure, send processed events to downstream services.

Code/Config Example:

lambda_client.update_event_source_mapping(
DestinationConfig={'OnSuccess': {'Destination': 'arn:aws:sqs:...:success-queue'}}
)

🔍 Interviewer Notes: Tests Lambda destinations.

❌ Common Mistakes:

  • Not configuring destinations
  • DLQ vs destinations
  • Missing IAM

Q273: How do you handle MSK rolling restarts?​

Technical Answer: Restart brokers one by one, ensure leader election completes before restarting next, and monitor under-replicated partitions.

Code/Config Example:

# Restart broker 1
kafka-server-stop.sh
kafka-server-start.sh server.properties

🔍 Interviewer Notes: Tests rolling restarts.

❌ Common Mistakes:

  • Restarting all brokers
  • Not checking leaders
  • High unavailability

Q274: How do you use MSK with Apache Spark?​

Technical Answer: Use Spark Streaming with Kafka source, configure Spark job with MSK bootstrap servers and authentication.

Code/Config Example:

val df = spark.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "broker:9098")
.option("subscribe", "orders")
.load()

🔍 Interviewer Notes: Tests Spark integration.

❌ Common Mistakes:

  • Wrong format
  • Not configuring auth
  • Checkpointing

Q275: How do you handle MSK broker disk full?​

Technical Answer: Increase EBS volume size, delete old data, or add brokers. Monitor disk usage via CloudWatch.

Code/Config Example:

# Increase volume size
kafka.update_broker_type(
BrokerNodeGroupInfo={'StorageInfo': {'EbsStorageInfo': {'VolumeSize': 2000}}}
)

🔍 Interviewer Notes: Tests disk full handling.

❌ Common Mistakes:

  • Not monitoring disk
  • Disk full outages
  • Not increasing size

Q276: How do you use MSK with Amazon OpenSearch?​

Technical Answer: Use MSK Connect OpenSearch sink connector, or Lambda to index MSK topic data into OpenSearch.

Code/Config Example:

connector_config = {
'connector.class': 'opensearch.OpenSearchSinkConnector',
'topics': 'orders'
}

🔍 Interviewer Notes: Tests OpenSearch integration.

❌ Common Mistakes:

  • Not using connector
  • Index mapping
  • IAM permissions

Q277: How do you handle MSK network partitions?​

Technical Answer: MSK brokers use quorum (ZooKeeper/KRaft) to handle network partitions, disable unclean leader election to avoid split-brain.

Code/Config Example:

admin_client.alter_configs([ConfigResource(
ConfigResourceType.TOPIC, 'orders',
configs={'unclean.leader.election.enable': 'false'}
)])

🔍 Interviewer Notes: Tests network partition handling.

❌ Common Mistakes:

  • Enabling unclean election
  • Not monitoring quorum
  • Split-brain

Q278: How do you use MSK with AWS IoT Core?​

Technical Answer: Use IoT Core rules to send device data to MSK via Lambda or IoT Rule action (Kafka action).

Code/Config Example:

# IoT Rule
SELECT * FROM 'iot/topic' WHERE temperature > 30 THEN send to Kafka topic 'high-temp'

🔍 Interviewer Notes: Tests IoT integration.

❌ Common Mistakes:

  • Not using IoT Rule
  • IAM permissions
  • Data format

Q279: How do you handle MSK producer batch expiry?​

Technical Answer: Increase linger.ms, reduce batch.size, or increase request.timeout.ms to avoid batch expiry before sending.

Code/Config Example:

producer = KafkaProducer(
linger_ms=100,
request_timeout_ms=30000
)

🔍 Interviewer Notes: Tests batch expiry.

❌ Common Mistakes:

  • Short timeout
  • Large batches
  • High expiry rate

Q280: How do you use MSK with Apache NiFi?​

Technical Answer: Use NiFi Kafka processors to produce/consume MSK topics, configure NiFi with MSK bootstrap servers and auth.

Code/Config Example:

# NiFi Kafka producer config
bootstrap.servers: broker:9098
security.protocol: SASL_SSL

🔍 Interviewer Notes: Tests NiFi integration.

❌ Common Mistakes:

  • Not using NiFi processors
  • Auth config
  • Data flow issues

Q281: How do you migrate MSK to KRaft mode?​

Technical Answer: Follow official MSK KRaft migration guide, test in non-prod first, and verify controller quorum.

Code/Config Example:

# KRaft migration step
kafka-storage.sh format --config server.properties --cluster-id <id>

🔍 Interviewer Notes: Tests KRaft migration.

❌ Common Mistakes:

  • Not testing
  • Quorum issues
  • Downtime

Q282: How do you handle MSK broker full GC?​

Technical Answer: Monitor broker GC metrics, increase heap size, or use smaller batch sizes to reduce memory pressure.

Code/Config Example:

# Monitor GC via JMX
cloudwatch.get_metric_statistics(Namespace='Kafka', MetricName='GcCount')

🔍 Interviewer Notes: Tests GC handling.

❌ Common Mistakes:

  • Not monitoring GC
  • Heap too small
  • Long GC pauses

Q283: How do you use MSK with AWS Backup?​

Technical Answer: MSK does not support AWS Backup directly, use MSK Connect to S3 then backup S3 with AWS Backup, or snapshot EBS volumes.

Code/Config Example:

# Backup S3 with AWS Backup
backup.create_backup_plan(BackupPlan={'BackupPlanName': 's3-backup'})

🔍 Interviewer Notes: Tests backup integration.

❌ Common Mistakes:

  • Expecting MSK Backup
  • Not backing up S3
  • EBS snapshot issues

Q284: How do you handle MSK topic retention overflow?​

Technical Answer: Monitor topic disk usage, increase retention time/size, or add brokers to spread data.

Code/Config Example:

admin_client.alter_configs([ConfigResource(
ConfigResourceType.TOPIC, 'orders',
configs={'retention.bytes': '10737418240'} # 10GB
)])

🔍 Interviewer Notes: Tests retention overflow.

❌ Common Mistakes:

  • Not monitoring
  • Retention too short
  • Disk full

Q285: How do you use MSK with Cloudera Data Platform?​

Technical Answer: Configure Cloudera Kafka client to connect to MSK, use IAM/SCRAM authentication, and manage topics via Cloudera.

Code/Config Example:

# Cloudera Kafka config
bootstrap.servers=broker:9098
security.protocol=SASL_SSL

🔍 Interviewer Notes: Tests Cloudera integration.

❌ Common Mistakes:

  • Not configuring auth
  • IAM permissions
  • Version mismatch

Q286: How do you handle MSK consumer commit failures?​

Technical Answer: Retry commits, use async commit with callback, or store offsets externally (DB) for exactly-once.

Code/Config Example:

consumer.commit_async(callback=lambda offsets, exception: print(f"Commit failed: {exception}"))

🔍 Interviewer Notes: Tests commit failures.

❌ Common Mistakes:

  • Not retrying
  • Sync commit blocking
  • Offset loss

Q287: How do you use MSK with Confluent Platform?​

Technical Answer: Configure Confluent components (Schema Registry, Connect) to use MSK as Kafka cluster, use IAM authentication.

Code/Config Example:

# Confluent Schema Registry config
kafkastore.bootstrap.servers=broker:9098

🔍 Interviewer Notes: Tests Confluent integration.

❌ Common Mistakes:

  • Not configuring auth
  • Version mismatch
  • Compatibility

Q288: How do you handle MSK broker CPU starvation?​

Technical Answer: Increase broker instance size, scale out brokers, or optimize producer/consumer throughput to reduce broker load.

Code/Config Example:

# Scale up broker
kafka.update_broker_type(BrokerNodeGroupInfo={'InstanceType': 'kafka.m5.2xlarge'})

🔍 Interviewer Notes: Tests CPU starvation.

❌ Common Mistakes:

  • Not monitoring CPU
  • Under-provisioning
  • High throughput

Q289: How do you use MSK with Google Cloud Pub/Sub?​

Technical Answer: Use Kafka Connect Pub/Sub connector, or Lambda to bridge MSK and Pub/Sub topics.

Code/Config Example:

connector_config = {
'connector.class': 'com.google.pubsub.kafka.connector.PubSubSinkConnector',
'topics': 'orders'
}

🔍 Interviewer Notes: Tests cross-cloud integration.

❌ Common Mistakes:

  • Not using connector
  • Auth issues
  • Data format

Q290: How do you handle MSK zero-downtime upgrades?​

Technical Answer: Use MSK rolling upgrades, test clients with new version, and monitor upgrade progress.

Code/Config Example:

kafka.update_cluster_kafka_version(TargetKafkaVersion='3.6.0')

🔍 Interviewer Notes: Tests zero-downtime upgrades.

❌ Common Mistakes:

  • Not testing
  • Rolling upgrade issues
  • Client incompatibility

Q291: How do you use MSK with Azure Event Hubs?​

Technical Answer: Use Kafka Connect Event Hubs connector, or mirror topics between MSK and Event Hubs using MirrorMaker 2.

Code/Config Example:

# Event Hubs Kafka endpoint
bootstrap_servers = 'my-namespace.servicebus.windows.net:9093'

🔍 Interviewer Notes: Tests Azure integration.

❌ Common Mistakes:

  • Not using Kafka endpoint
  • Auth issues
  • Cross-cloud costs

Q292: How do you handle MSK audit log analysis?​

Technical Answer: Use CloudWatch Logs Insights to query audit logs, search for specific operations, and create dashboards.

Code/Config Example:

# CloudWatch Logs Insights query
fields @timestamp, operation, user
| filter operation = 'CreateTopic'

🔍 Interviewer Notes: Tests audit log analysis.

❌ Common Mistakes:

  • Not enabling audit logs
  • Not analyzing
  • Missing alerts

Q293: How do you use MSK with Databricks?​

Technical Answer: Use Databricks Kafka connector to read/write MSK topics, configure with MSK bootstrap servers and auth.

Code/Config Example:

df = spark.read.format("kafka") \
.option("kafka.bootstrap.servers", "broker:9098") \
.option("subscribe", "orders") \
.load()

🔍 Interviewer Notes: Tests Databricks integration.

❌ Common Mistakes:

  • Not using connector
  • Auth config
  • Checkpointing

Q294: How do you handle MSK broker unresponsive?​

Technical Answer: Check broker metrics, restart broker, or replace broker via MSK API. Investigate root cause (GC, disk full, network).

Code/Config Example:

kafka.update_broker_count(TargetNumberOfBrokerNodes=3) # Replace broker

🔍 Interviewer Notes: Tests unresponsive broker.

❌ Common Mistakes:

  • Not investigating root cause
  • Delayed restart
  • Data loss

Q295: How do you use MSK with Snowflake?​

Technical Answer: Use MSK Connect Snowflake sink connector, or load data from S3 (via S3 sink) into Snowflake.

Code/Config Example:

connector_config = {
'connector.class': 'com.snowflake.kafka.connector.SnowflakeSinkConnector',
'topics': 'orders'
}

🔍 Interviewer Notes: Tests Snowflake integration.

❌ Common Mistakes:

  • Not using connector
  • IAM permissions
  • Data format

Q296: How do you handle MSK consumer rebalance storms?​

Technical Answer: Use cooperative rebalancing, increase session.timeout.ms, or reduce max.poll.interval.ms to avoid frequent rebalances.

Code/Config Example:

consumer = KafkaConsumer(
partition_assignment_strategy='CooperativeStickyAssignor',
session_timeout_ms=45000
)

🔍 Interviewer Notes: Tests rebalance storms.

❌ Common Mistakes:

  • Not using cooperative
  • Low session timeout
  • Frequent consumer joins

Q297: How do you use MSK with MongoDB?​

Technical Answer: Use MSK Connect MongoDB sink/source connector, or change streams to write to MSK topics.

Code/Config Example:

connector_config = {
'connector.class': 'com.mongodb.kafka.connect.MongoSourceConnector',
'connection.uri': 'mongodb://mongo:27017'
}

🔍 Interviewer Notes: Tests MongoDB integration.

❌ Common Mistakes:

  • Not using connector
  • Auth issues
  • Change stream config

Q298: How do you handle MSK producer queue full?​

Technical Answer: Increase buffer.memory, reduce max.block.ms, or slow down producer send rate to avoid queue full errors.

Code/Config Example:

producer = KafkaProducer(
buffer_memory=134217728, # 128MB
max_block_ms=60000
)

🔍 Interviewer Notes: Tests queue full handling.

❌ Common Mistakes:

  • Default buffer too small
  • Not handling errors
  • Producer blocking

Q299: How do you use MSK with Apache Pulsar?​

Technical Answer: Use Kafka protocol handler for Pulsar, or bridge topics between MSK and Pulsar using connectors.

Code/Config Example:

# Pulsar Kafka protocol handler
messagingProtocol=kafka

🔍 Interviewer Notes: Tests Pulsar integration.

❌ Common Mistakes:

  • Not using protocol handler
  • Compatibility issues
  • Cross-cluster sync

Q300: How do you handle MSK end-of-life Kafka versions?​

Technical Answer: Upgrade to supported Kafka version before EOL, test compatibility, and monitor MSK version support announcements.

Code/Config Example:

kafka.update_cluster_kafka_version(TargetKafkaVersion='3.6.0')

🔍 Interviewer Notes: Tests EOL handling.

❌ Common Mistakes:

  • Not upgrading before EOL
  • Compatibility issues
  • Downtime

Section 4: Extended Advanced Questions Summary​

Key Advanced Concepts Checklist:​

  • Schema Registry management
  • Multi-region replication
  • Compliance and audit logging
  • Migration and upgrades
  • Advanced stream processing (Kafka Streams, ksqlDB)
  • Cross-cloud integration
  • Performance tuning
  • Disaster recovery

Final Interview Quick Reference:​

ConceptKey Point
Exactly-OnceIdempotent producers + transactions
Schema EvolutionBackward compatibility by default
Multi-RegionMirrorMaker 2 for replication
SecurityIAM + TLS + Encryption at rest
MonitoringCloudWatch + Prometheus + JMX

Additional Core Questions Summary​

Key MSK/Kafka Concepts Checklist:​

  • Cluster types (provisioned vs serverless)
  • Producer and consumer patterns
  • Topics and partition strategies
  • Consumer lag monitoring
  • MSK Connect for data ingestion
  • IAM authentication and security
  • Multi-region replication (MirrorMaker 2)
  • Performance tuning (batching, compression)
  • Monitoring with CloudWatch

Interview Quick Reference:​

ConceptKey Point
Partition CountMore = more parallelism, but rebalancing overhead
Replication Factor3 for production (can tolerate 2 failures)
Consumer LagDifference between latest offset and committed offset
min.insync.replicasMinimum replicas for acks=all writes
RetentionBased on time (retention.ms) or size (retention.bytes)