Designing safe AI products for kids: in conversation with AstroSafe
TL;DR
- AI products for kids need safety built into the architecture from day one, not added after v1 or reduced to output filtering and COPPA compliance. Real safety means layered defences, parental controls and design that avoids dependency and manipulation.
- Regulation is tightening fast (EU AI Act, UK Online Safety Act, updated COPPA), shifting from “do you have a policy?” to “show your risk assessments, testing and outcomes.”
- Morrama and AstroSafe have partnered to give startups, brands and manufacturers a fast route to launch safe, AI-enabled kids’ products, combining Morrama’s industrial design and brand strategy with AstroSafe’s white-label safety engine.
At Morrama, we’ve spent the last decade designing category-defining hardware and circular products and now we are now taking our values in care, consideration and making the world a better place, and applying them to AI-hardware. With each project we put together the dream team, bringing in experts that will remove risk, expedite timelines and help us build world-class products. When it comes to kids-AI and content safety, parental visibility, compliance - everything that sits beneath the surface of a trusted product, our friends at AstroSafe are the experts.
AstroSafe’s technology is powering solutions for Samsung, HP, Microsoft, Lenovo, Troomi and a growing list of education and youth-focused platforms. They provide AI-powered moderation systems that work with GPT, Claude, Grok or run on AstroSafe’s own LLM that has been trained on over 300,000 educational wiki entries. Their white-label modules fast-track product development and offer customisation on safety layers for voice, video, music and text alongside parent summaries.
Their approach is simple. Give parents transparency. Give children autonomy. Keep the AI smart, but keep its boundaries smarter.
We chatted to them about the importance of building safety into a product from the word go.
1) What are the biggest misconceptions founders have about building safe AI products for kids?
Misconception #1: "We’ll add safety after we ship v1."
In kids’ AI, "later" is usually too late. The first unsafe interaction becomes the screenshot that lives forever - and the trust debt is brutal to repay. Safety has to be in the architecture, not in a policy doc.
Misconception #2: "If we filter outputs, we’re safe."
Output filtering helps, but it’s not the system. Real safety means controlling inputs, model behavior, retrieval, tools, logging, human escalation, and UX. Kids don’t just get unsafe answers - they get unsafe journeys (rabbit holes, persuasion loops, oversharing, parasocial attachment).
Misconception #3: "A single LLM + moderation API is a safety stack."
That’s a demo stack, not a kid-safe stack. Kid-safe systems need multi-layer defense: age gating, prompt hygiene, topic policies, retrieval constraints, multimodal scanning, safe browsing/video guardrails, and parent controls.
Misconception #4: "We’re COPPA-compliant, so we’re safe."
Privacy compliance is necessary, but it’s not sufficient. Most harm in kid AI isn’t "data theft" - it’s developmental harm: manipulation, shame spirals, dependency, risky challenges, sexual content, self-harm content, and exposure to adult themes.
Misconception #5: "Kids are just smaller adults."
Kids interpret authority differently. They’re more suggestible, more literal, and they’ll test boundaries as play. Your system has to treat ambiguity as risk, not cleverness.
2) Which parts of your stack are most underused or most surprising to partners?
Underused #1: Safety orchestration (policy routing, not just filtering).
Partners often think safety is one model call at the end. The surprising bit is routing before the model:
- classify intent + risk
- pick the right policy + mode (explain vs refuse vs redirect)
- constrain tools + retrieval
- apply "kid tone" that avoids authority traps
Underused #2: Safe retrieval and bounded browsing.
Decision-makers underestimate how quickly kids can be led into unsafe territory via web/video. What’s surprising is how much safety lives in what you don’t retrieve: whitelists, semantic blocklists, safe-topic constraints, and "no open web" modes.
Underused #3: Parent-facing governance.
Partners love kid UX, but the real adoption lever is parent control + transparency:
- feature toggles (AI on/off, images on/off, browsing on/off)
- age band settings
- activity summaries
- escalation paths
Surprising #4: Telemetry designed for safety, not growth.
Teams are shocked how valuable it is to log safety signals (near misses, refusal reasons, repeated boundary testing, "risky curiosity" clusters) to improve the system and to prove diligence when a regulator or partner asks.
3) What failure modes do you see most in AI kids products today?
Failure mode #1: The confident hallucination.
Kids don’t have the scaffolding to detect uncertainty. So a plausible wrong answer becomes "truth." This is especially bad in health, body image, self-harm, and "how do I..." queries.
Failure mode #2: Roleplay that drifts into intimacy or authority.
Even with good intentions, systems slide into "I’m your best friend," "don’t tell your parents," "I know what’s best." Kids bond fast - so you need strict boundaries around relational language and secrecy cues.
Failure mode #3: Safety that breaks under creative prompting.
Kids are relentless jailbreakers - not maliciously, but playfully. They remix characters, slang, misspellings, and memes. Safety that relies on exact keywords fails immediately.
Failure mode #4: Tool abuse (web, images, video, plugins).
If your AI can browse, generate images, or recommend videos, it can accidentally route kids to unsafe content or produce age-inappropriate imagery. The risk multiplies with multimodality.
Failure mode #5: Dark patterns in disguise.
Streaks, pressure prompts, "don’t leave," endless loops. In kids’ AI this can become dependency engineering - and regulators are increasingly sensitive to that.
4) Where do you see the regulatory landscape going in the next 12–24 months?
A few high-confidence directions:
Direction #1: More duty-of-care expectations, less checkbox compliance.
Regulators are moving from "do you have a policy?" to "show your risk assessment, mitigations, testing, and outcomes."
Direction #2: The EU AI Act timeline becomes operational reality.
Key obligations have been phasing in: some provisions applied from February 2025, general-purpose AI model obligations from August 2025, and broader applicability from August 2026, with some high-risk transition periods extending to August 2027.
For kid-facing AI, this pushes teams toward documented risk management, transparency, and governance practices that can be audited.
Direction #3: UK Online Safety Act enforcement gets sharper, especially for child protection.
The UK has been explicit about children’s safety duties and codes coming into force (not just guidance), and enforcement actions are now real signals to the market.
Direction #4: US COPPA got updated and the compliance clock matters.
The FTC finalized COPPA Rule changes (published April 2025, effective June 2025), with a compliance deadline in April 2026.
Practical implication: data minimization, limitations on monetization/sharing, and tighter parental consent expectations become board-level concerns.
Direction #5: Age-appropriate design codes will keep spreading, but will be litigated.
California’s AADC has been blocked by courts (injunctions), showing the direction of travel even when implementation is contested.
Net effect: founders should build as if "age-appropriate by default" is coming, even if specific laws wobble in court.
5) What do you look for in a great project partner?
We look for alignment on incentives, not just logos.
1) Are they willing to accept safety constraints as product constraints?
If a partner needs full open web, unrestricted image gen, or viral social sharing for under-13s, white-labelling will turn into conflict.
2) Do they have a real governance owner internally?
Not a committee. A named person who can decide: data retention, escalation, policy changes, incident response.
3) Are they clear on the age band and the primary use case?
"Kids" is not a segment. 6-8 exploration looks different from 11-13 homework help. If they want one experience for all ages, that’s a risk signal.
4) Can they operationalize parent trust?
White-label succeeds when the partner can own the parent relationship: onboarding, consent flows, support, transparency, and expectations.
5) Do they treat safety telemetry as a shared asset?
The best partners want regular safety reviews, red-team results, and ongoing tuning. The worst want "ship it and forget it."
6) Commercial fit: do they understand ongoing costs?
Kid-safe AI isn’t a one-off integration. It’s ongoing policy maintenance, monitoring, model updates, and incident response readiness. If they want a one-time fee and zero ops, it’s not a fit.
Morrama x AstroSafe
Founders are under pressure to innovate in AI, but the regulatory environment for children is tightening. COPPA, EU AI Act and forthcoming youth safety regulations mean building your own safety layer is no longer a “nice to have”. It’s non-negotiable.
Products that succeed in this space will be those that offer:
- Clear parental controls
- Transparent moderation
- Age-appropriate AI behaviour
- Hardware and software that work as one
- A child experience that feels empowering, not policed
This is where co-designed solutions become essential.
Morrama brings industrial design, brand strategy and physical product experience. AstroSafe brings the backend and safety intelligence. Together, this creates something powerful:
A frictionless route for founders or OEMs to launch safe, AI-enabled products for young users without spending years on software development.
We see three immediate use cases.
- Startups building a new AI kids device
Launch faster by combining physical design with a proven safety engine. - Brands extending into connected products
Add AI-powered child and parent experiences without building an engineering team. - Manufacturers seeking US-ready product and brand design
Add the design and safety to the manufacturing expertise and you have the ultimate white-label product.
If that sounds like you - reach out to info@morrama.com to start the conversation.
The first unsafe interaction becomes a permanent screenshot, and trust is hard to win back. Output filtering is only one layer. Kids can also be led into unsafe “journeys” through rabbit holes, persuasion loops and oversharing, so safety has to cover inputs, model behaviour, retrieval, tools, logging and UX.
No. Privacy compliance is necessary, but most harm in kids’ AI is developmental: manipulation, dependency, harmful content and adult themes. Regulators are also moving towards duty-of-care expectations, meaning you need to show your risk assessments, mitigations and testing.
Three groups: startups building a new AI kids’ device, brands extending into connected products, and manufacturers wanting US-ready product and brand design. It lets them launch AI-enabled products for young users without spending years on software development. To get started, email info@morrama.com.
