Hugging Face's logo
Aliasgar Khimani's picture
๐Ÿค Open to Collab

Aliasgar Khimani

NovusEdge
1
mayafree's profile picture Trybook's profile picture
ยท

AI & ML interests

None yet

Recent Activity

repliedto ginigen-ai's post about 1 month ago
๐Ÿง  Does your LLM know when it's about to be wrong? Most leaderboards measure accuracy. We measure metacognition โ€” whether a model catches its own errors. Benchmark + leaderboard + adapters, all open. ๐ŸŽ‰ The surprise: even a K-AI #1 model (JGOS-31B-Citizen) is the strongest on multiple-choice traps (trap_rate 0.005 โ€” ~2 misses in 400) yet blind to its own free-form mistakes (self-confidence AUROC = 0.5, pure random). A tiny base-frozen adapter recovers that signal. Two independent axes (never compared across a row): โ‘  trap_rate โ€” does it fall for tempting trap options? (lower = stronger) โ‘ก adapter gain ฮ” โ€” how much a lightweight adapter catches errors the model itself misses. (higher = more adapter value) What's open: ๐Ÿ“Š 300+100 trap problems (each with a hidden trap + TICOS type) ๐Ÿ† 24-model leaderboard ๐Ÿงฉ 11 per-model adapters โ€” adapters, NOT fine-tunes (base stays frozen; the adapter just reads the hidden state โ†’ P(wrong)) Submit any HF model โ†’ auto-scored daily at 09:00 KST and added to the board. ๐Ÿ† Leaderboard โ†’ https://huggingface.co/spaces/ginigen-ai/Metacognition-Leaderboard-Space ๐Ÿ“Š Benchmark โ†’ https://huggingface.co/datasets/ginigen-ai/Metacognition-Bench ๐Ÿงฉ Adapters โ†’ https://huggingface.co/collections/FINAL-Bench/metacognition-adapters-6a42c032e6beb803dd032961 ๐Ÿ“Š Article โ†’ https://huggingface.co/blog/ginigen-ai/metacognition Benchmark by ginigen-ai ยท Adapters by FINAL-Bench (Darwin/Chimera platform + AETHER metacognition tech).
reacted to ginigen-ai's post with ๐Ÿ”ฅ about 1 month ago
๐Ÿง  Does your LLM know when it's about to be wrong? Most leaderboards measure accuracy. We measure metacognition โ€” whether a model catches its own errors. Benchmark + leaderboard + adapters, all open. ๐ŸŽ‰ The surprise: even a K-AI #1 model (JGOS-31B-Citizen) is the strongest on multiple-choice traps (trap_rate 0.005 โ€” ~2 misses in 400) yet blind to its own free-form mistakes (self-confidence AUROC = 0.5, pure random). A tiny base-frozen adapter recovers that signal. Two independent axes (never compared across a row): โ‘  trap_rate โ€” does it fall for tempting trap options? (lower = stronger) โ‘ก adapter gain ฮ” โ€” how much a lightweight adapter catches errors the model itself misses. (higher = more adapter value) What's open: ๐Ÿ“Š 300+100 trap problems (each with a hidden trap + TICOS type) ๐Ÿ† 24-model leaderboard ๐Ÿงฉ 11 per-model adapters โ€” adapters, NOT fine-tunes (base stays frozen; the adapter just reads the hidden state โ†’ P(wrong)) Submit any HF model โ†’ auto-scored daily at 09:00 KST and added to the board. ๐Ÿ† Leaderboard โ†’ https://huggingface.co/spaces/ginigen-ai/Metacognition-Leaderboard-Space ๐Ÿ“Š Benchmark โ†’ https://huggingface.co/datasets/ginigen-ai/Metacognition-Bench ๐Ÿงฉ Adapters โ†’ https://huggingface.co/collections/FINAL-Bench/metacognition-adapters-6a42c032e6beb803dd032961 ๐Ÿ“Š Article โ†’ https://huggingface.co/blog/ginigen-ai/metacognition Benchmark by ginigen-ai ยท Adapters by FINAL-Bench (Darwin/Chimera platform + AETHER metacognition tech).
repliedto ginigen-ai's post about 1 month ago
๐Ÿง  Does your LLM know when it's about to be wrong? Most leaderboards measure accuracy. We measure metacognition โ€” whether a model catches its own errors. Benchmark + leaderboard + adapters, all open. ๐ŸŽ‰ The surprise: even a K-AI #1 model (JGOS-31B-Citizen) is the strongest on multiple-choice traps (trap_rate 0.005 โ€” ~2 misses in 400) yet blind to its own free-form mistakes (self-confidence AUROC = 0.5, pure random). A tiny base-frozen adapter recovers that signal. Two independent axes (never compared across a row): โ‘  trap_rate โ€” does it fall for tempting trap options? (lower = stronger) โ‘ก adapter gain ฮ” โ€” how much a lightweight adapter catches errors the model itself misses. (higher = more adapter value) What's open: ๐Ÿ“Š 300+100 trap problems (each with a hidden trap + TICOS type) ๐Ÿ† 24-model leaderboard ๐Ÿงฉ 11 per-model adapters โ€” adapters, NOT fine-tunes (base stays frozen; the adapter just reads the hidden state โ†’ P(wrong)) Submit any HF model โ†’ auto-scored daily at 09:00 KST and added to the board. ๐Ÿ† Leaderboard โ†’ https://huggingface.co/spaces/ginigen-ai/Metacognition-Leaderboard-Space ๐Ÿ“Š Benchmark โ†’ https://huggingface.co/datasets/ginigen-ai/Metacognition-Bench ๐Ÿงฉ Adapters โ†’ https://huggingface.co/collections/FINAL-Bench/metacognition-adapters-6a42c032e6beb803dd032961 ๐Ÿ“Š Article โ†’ https://huggingface.co/blog/ginigen-ai/metacognition Benchmark by ginigen-ai ยท Adapters by FINAL-Bench (Darwin/Chimera platform + AETHER metacognition tech).
View all activity

Organizations

Engrammic's profile picture