How to analyse public submissions without losing the source

Why does submission analysis have to be traceable?

Because a theme in a policy, by-law or report is only defensible if you can show which submissions it came from. In a public process, someone will eventually ask what people said, who said it, and whether it was considered.

Submissions arrive through unlike channels (letters, online forms, survey comments, petitions, hearing notes), run from one line to fifty pages, and include campaign templates. The job is not only to find themes. It is to compare responses in a way that holds up when a reviewer, a portfolio committee or a court asks where a finding came from.

This guide is the method page for the qualitative evidence topic. It sets out the steps we use, based on the Framework Method, with the control layer that keeps each claim linked to its source.

What does South African law expect of public participation?

South African law requires legislatures and municipalities to involve the public, and requires administrators to consider comments in certain decisions. None of these provisions prescribes an analysis method, but each one assumes the record can show that public input was received and taken into account.

The main provisions, checked against primary sources on 12 October 2026:

The Constitution was approved by the Constitutional Court on 4 December 1996 and took effect on 4 February 1997. Chapter 4 of the Municipal Systems Act came into operation on 1 March 2001.

The Constitutional Court has set out how the constitutional duty is tested. In Doctors for Life International v Speaker of the National Assembly (2006), it held that the question is whether what the legislature did was reasonable in all the circumstances, and that the duty will often require giving people a meaningful opportunity to be heard. The Court declared that Parliament had failed to facilitate public involvement, as section 72(1)(a) requires, before passing two health statutes, and declared both invalid.

For an analysis team, the point is practical. If a process is challenged, the evidence that submissions were considered is the analysis itself: the intake log, the coding, the matrix and the trail back to each submission. That record has to exist before anyone asks for it.

What is the Framework Method, and why does it suit submissions?

The Framework Method is a structured way to analyse qualitative material in a matrix, with cases as rows and codes as columns, and summarised data in each cell (Gale et al., 2013). It was developed at the National Centre for Social Research in the UK in the late 1980s for large-scale policy research.

It suits public submissions for three reasons:

• It compares across cases without losing them. Each submission (or respondent) stays a row, so you can read across one submission or down one theme.
• It leaves an audit trail. Gale et al. note that the method gives a clear trail from raw data to final themes, including the quotes used to illustrate them.
• It works in teams. The method was written up for multi-disciplinary teams where not everyone is a trained qualitative researcher, provided an experienced analyst leads.

Gale et al. describe seven stages: transcription, familiarisation, coding, developing a working analytical framework, applying it, charting data into the framework matrix, and interpreting the data. The steps below adapt them to submissions, where most material is already written and intake is the bigger problem.

How do you analyse public submissions step by step?

Set up the source controls first, then build and test the codebook, then code, chart and draft. Most late problems come from coding before the intake was stable.

1. Define the output and keep the channels separate

Decide what the analysis has to produce (a consultation report, a thematic chapter, a response matrix, a briefing) and which response channels feed it. Keep letters, online submissions, survey comments, petitions and hearing records visible as distinct channels from the start.

Answer these before coding:

• What output has to be produced, and for whom?
• Which channels feed it?
• Which respondent groups need to be compared?
• What caveats will have to be stated (campaign responses, uneven reach, unweighted counts)?

2. Build the intake register and source IDs

Give every submission a stable ID before anyone codes it, and record enough metadata to compare groups later. This register is the control layer that lets every coded point travel back to a source. (How to keep that link through to the final report is covered in our guide to source traceability.)

Decide the locator rule now (page and paragraph, question and response number, transcript line), so the charting stage does not have to retrofit it.

3. Clean the set before coding

Remove noise before it shapes the codebook, and record every cleaning decision. Check for blank responses, duplicates, the same submission logged twice under different names, broken files, and campaign or template responses. Campaign responses are evidence of organised support and should be counted and reported as such, not silently merged or deleted.

4. Read a cross-section and write memos

Read across channels, respondent types and positions before building any codes. Keep short memos on recurring issues, outliers, differences between groups, language worth quoting, and anything that could affect interpretation. The Cochrane Handbook's chapter on qualitative evidence notes that extraction is not linear: teams move back and forth between stages, and must preserve the context of the source material to avoid misreading it (Noyes et al., Cochrane Handbook version 6.5, 2024). Memos make those early decisions visible instead of leaving them in one person's head.

5. Draft the codebook

Turn the read-through into codes the whole team can apply the same way. In consultation work, the reporting questions usually supply the starting structure, and the material adds and sharpens codes as you go. Each code needs a definition, inclusion and exclusion rules, and any qualifiers.

Code name:     service_access_barrier
Definition:    People cannot reach or use a service because of distance,
               cost, transport, documentation or physical access.
Include when:  The respondent describes a practical barrier to using
               or attending the service.
Exclude when:  The complaint is about service quality after access
               was secured.
Qualifier:     Record the barrier type: transport cost, travel time,
               disability access or documentation.
Version:       v1.2, changed 3 March: transport cost split out as a qualifier

Version the codebook. Once codes start changing, record what changed and why.

6. Pilot the coding and align the team

Code a small sample from several channels, compare results between coders, and fix the codebook before the full pass. Look for overlapping codes, codes that are too broad or too narrow, missing categories, and the same issue coded two ways.

For example, a survey comment reads: "People in Ward 4 stop going to the clinic because it is too far and taxi fares are too high." One coder applies distance barrier and transport cost; another applies service access barrier. The team keeps service_access_barrier as the code, adds a transport-cost qualifier, and updates the definition. The goal is not perfect agreement statistics. It is to find drift while the codebook can still change.

7. Code the full set with exact locators

Every coded point carries its submission ID, an exact locator, the excerpt or a close paraphrase, the channel and group metadata, and the codebook version. A coded record should say what was raised, where, and by whom, not only which theme it belongs to.

Strong quotes found here can go straight into a quote bank, already linked to their source.

8. Chart into a framework matrix and compare

Summarise the coded material into a matrix of cases by codes, with a locator in every cell. This is where the submission set becomes something you can compare: one group against another, one channel against another, one place against another, and outlier positions that should stay visible.

Charting is summarising, not pasting. A cell should tell a reviewer the point, its context, and where to check it.

9. Draft from the matrix and test the trail

Write findings from the matrix, not from the raw files, and check the route back to source before sign-off. Each finding should answer four questions: what is the pattern, which coded records support it, which groups or channels it applies to, and where a reviewer can verify it.

Finding:   Transport cost was a recurring access barrier in rural responses.
Matrix:    Rural respondents x service_access_barrier (transport-cost qualifier)
Records:   SUB-LET-014, SUR-088, HRG-012
Locators:  Letter p. 3 paras 2-3; survey Q14 response 088;
           hearing notes p. 6 lines 11-18
Caveats:   Unweighted; rural respondents were a minority of the set
QA:        Locators spot-checked; wording matches coded evidence

Should letters, surveys, petitions and hearings be analysed together?

Read them against each other, but record and report them as separate channels. A petition with many signatures, a long technical letter and a one-line survey comment are different kinds of evidence, and counting them in one pool hides that.

In practice:

• Keep channel as a field on every record and every matrix row.
• Report counts by channel and by respondent type, and say what the counts do and do not mean.
• Treat campaign and template responses as one position with a count of supporters, and say so in the method note.
• Do not present unweighted counts as a measure of public opinion. Submissions come from people who chose to respond.

Where does AI fit in submission analysis?

AI can help with retrieval and first-pass work once the intake register, codebook and matrix are stable. It should not decide what the public said. People review its output and own every finding.

Useful, checkable tasks include finding duplicates and template responses, suggesting candidate codes for a coder to accept or reject, retrieving every passage on a topic, and drafting summaries that are then checked against the source. Each depends on source IDs and locators already being in place. Without them, AI output may read well and still be impossible to verify.

The White Paper on Local Government submissions case study shows this order in practice: intake, claim-level coding and synthesis came first, AI was used only for retrieval on approved material, and policy decisions stayed with the drafting team.

How does the analysis feed what comes next?

The coded records and matrix should feed every later product directly, so nobody re-reads the submissions to write the report. Three outputs usually follow:

• A response record. When a draft is published for comment, each comment needs a documented response and decision. Our guide to the public consultation response matrix sets out the fields.
• Ongoing feedback. Where public input is continuous rather than a single round, as with ward-level service monitoring, the same intake and coding discipline applies. See community-led monitoring.
• Interview and case-study evidence. Many processes add interviews or focus groups to written submissions. The coding approach is the same, with transcripts as the source; see how to turn interviews into report-ready findings.

When is a spreadsheet enough?

A spreadsheet works when the set is small, one person can read and check every submission, and no public response or formal decision record is required. Move to a structured database when volume is high, formats are mixed, several analysts code in parallel, or the output must stand up to public scrutiny.

When the structured route is needed, what makes it work is that submissions, codes, quotes, findings and later comments sit in one linked record. If you need that in place for a live process, we offer a source-linked submissions database, built and handed over to your team.

Common mistakes

• Coding before logging. If the intake register is weak, the analysis cannot be checked later. Log and ID everything first.
• Merging channels too early. Once letters, surveys and petitions are blended without a channel field, you cannot explain what the evidence shows.
• Deleting campaign responses. They are a position with a count. Record them as such and say how they were treated.
• Codes without definitions. A label with no inclusion or exclusion rule will be applied differently by each coder.
• Skipping the pilot. Drift found after the full pass means recoding.
• Drafting from raw files. If the report is written from memory and the original documents, the matrix stops being the record and the trail breaks.