How to Analyze Survey Responses with ChatGPT (Prompts and Pitfalls)

Quick answer: ChatGPT can help analyze open-ended survey responses if you use it in steps: ask it to draft a codebook from a sample, have it code responses in batches with response IDs, then count the codes yourself in a spreadsheet and check every quote against the original data. Do not paste personal data, and do not trust its counts or percentages without checking.

ChatGPT is a fast way to get a first look at survey comments. It is also easy to use badly: ask "what are the main themes?" and you get a plausible summary with numbers you cannot verify. The workflow below keeps the speed and removes most of the risk.

Before you start: prepare the data

1. Remove personal data. Delete names, emails, phone numbers and any detail that identifies a person. Check your company's AI policy and your ChatGPT plan's data settings before pasting customer or employee data.
2. Add a response ID to every row, such as R001, R002. You will need it to check the output.
3. Remove blanks and non-answers such as "n/a" and count them separately.
4. Keep one question per analysis.

Step 1: Draft a codebook from a sample

Paste 50 to 100 responses and ask for a codebook, not a summary:

You are a survey analyst. Below are answers to the question
"What could we improve?" from customers of an online store.

Create a codebook of 8 to 12 codes. For each code give:
- a short name
- a one-sentence definition
- what to include and what to exclude
- one example response ID that fits it

Do not summarize or count yet. Responses:
R001: ...
R002: ...

Read the codebook critically. Merge overlapping codes, split vague ones and rewrite unclear definitions. This is your codebook now, not the model's.

Step 2: Code responses in batches

Give the final codebook back and code in batches of 50 to 100 responses:

Using only the codebook below, code each response.
A response can have more than one code. Use "Other" if nothing fits
and "Non-answer" if the response has no content.

Return a table with exactly one row per response ID:
| Response ID | Codes |

Do not skip, merge or invent response IDs.

Codebook:
...

Responses:
R101: ...

Check that the number of rows in the output matches the batch. Missing or merged rows are the most common error.

Step 3: Count in a spreadsheet, not in the chat

Paste each batch's table into a spreadsheet and count the codes there. Language models are unreliable at counting and calculating percentages, especially across several batches. With codes in a spreadsheet you can use COUNTIF and pivot tables, and the numbers are exact. Our Excel guide shows the formulas.

Step 4: Spot-check the coding

Pick 20 random responses and code them yourself. If you agree with the model on fewer than about 8 in 10, tighten the codebook definitions and re-run the affected batches.

Step 5: Pick quotes from the original data

Ask for response IDs, not quotes:

For the code "Delivery speed", list the 5 response IDs that best
represent it. Return only the IDs.

Then copy the quotes from your original file. This avoids paraphrased or invented quotes appearing in your report.

What ChatGPT is good and bad at for survey analysis

Good at	Weak at
Drafting a codebook quickly	Counting and percentages
Suggesting names for themes	Consistency across batches and sessions
Summarizing a theme once you have its responses	Reproducibility: the same prompt can give different themes
Spotting ideas you missed	Proving which responses sit behind a number

When to use a dedicated tool instead

Consider a dedicated thematic analysis tool when:

• You have more than a few hundred responses
• You need counts you can defend in front of stakeholders
• You will repeat the analysis next month or next quarter
• You need to click a theme and see every response behind it

Apercu is built for this: upload a CSV or Excel file and it assigns every response to a named theme, with counts, percentages and examples you can trace. It does not use your content to train AI models. You can try it without an account.

FAQ

Can ChatGPT analyze survey data?

It can help with open-ended survey data, especially drafting a codebook and coding small batches. For reliable results, have it return codes per response ID, count the codes in a spreadsheet and check quotes against the original responses.

What is a good ChatGPT prompt for thematic analysis?

Ask for a codebook with definitions first, then ask it to code responses in batches into a table with one row per response ID using only that codebook. Avoid prompts like "summarize the main themes", which produce unverifiable summaries.

Is it safe to put survey responses into ChatGPT?

Remove personal and identifying information first, and follow your organization's AI policy. Check the data settings of your ChatGPT plan, since data handling differs between plans.

How many responses can ChatGPT analyze at once?

Modern models accept long inputs, but accuracy drops on long lists. Batches of 50 to 100 responses are easier to check and lead to fewer skipped or merged rows.