Skymizer Unveils HTX301 AI Inference Accelerator

Skymizer Targets AI Inference Bottleneck With New HTX301 Chip

Skymizer, a Taiwan-based AI IP company, has unveiled its HTX301 inference decode-phase accelerator chip, designed to work alongside GPUs to dramatically speed up large language model inference. The most striking specification: a single PCIe AIC (add-in card) can integrate 6 HTX301 chips with a combined 384GB of memory, enough to run 700-billion-parameter models locally — all while consuming just 240 watts of power.

The announcement, reported on May 7, positions Skymizer as a specialized player in the rapidly evolving AI inference hardware market. Rather than competing head-to-head with NVIDIA or AMD on general-purpose GPU compute, the company is carving out a niche in the decode phase of inference — a critical bottleneck that determines how quickly AI models generate output tokens.

Key Facts at a Glance

• Product: HTX301 inference decode accelerator chip
• Memory: 384GB per single PCIe AIC (6 chips per card)
• Power consumption: 240W per card
• Model support: Capable of running 700B-parameter models locally
• Architecture: Built on Skymizer's proprietary LISA instruction set
• Platform: HyperThought hardware-software co-design ecosystem
• Deployment: Supports both SoC and PCIe AIC form factors

Why the Decode Phase Matters for AI Inference

To understand why the HTX301 matters, it helps to break down how large language model inference actually works. The process consists of 2 distinct phases: prefill and decode.

During the prefill phase, the model processes the entire input prompt in parallel. This stage is compute-intensive and plays to the strengths of traditional GPUs with their massive parallel processing capabilities. The decode phase, however, is fundamentally different — it generates output tokens one at a time, sequentially, making it memory-bandwidth-intensive rather than compute-intensive.

This architectural mismatch means that expensive, power-hungry GPUs often sit underutilized during the decode phase. They are essentially overpowered for the task at hand, wasting energy and compute resources on a workload that demands fast memory access above all else. Skymizer's approach separates these 2 workloads, letting GPUs handle what they do best (prefill) while offloading decode to purpose-built hardware.

The result, according to Skymizer, is improved utilization of the overall compute system and significantly better energy efficiency. In a world where data center power consumption is becoming a first-order constraint, this kind of workload-specific optimization could prove transformative.

Inside the HTX301: Architecture and Design Philosophy

The HTX301 is built on Skymizer's HyperThought platform, which the company describes as a hardware-software co-design ecosystem. At its core, the chip uses a proprietary instruction set architecture called LISA (Language Instruction Set Architecture), purpose-built for bandwidth-intensive AI workloads.

Unlike general-purpose GPU architectures from NVIDIA (CUDA) or AMD (ROCm), LISA is specifically optimized for the sequential, memory-bound nature of token generation. This specialization allows Skymizer to make architectural tradeoffs that would be impossible in a general-purpose design — prioritizing memory bandwidth and capacity over raw floating-point throughput.

The scalable design supports 2 deployment form factors:

• SoC integration: For embedded and edge deployments where the HTX301 IP can be incorporated directly into custom silicon
• PCIe AIC: A standard add-in card format that can be installed in existing server infrastructure alongside GPUs

The PCIe AIC form factor is particularly noteworthy. By packing 6 HTX301 chips onto a single card with 384GB of total memory, Skymizer enables organizations to run some of the largest publicly available AI models — including those in the 700B-parameter class — without requiring the multi-GPU, multi-node setups that such models typically demand.

240W Power Draw Challenges GPU-Only Inference Setups

Perhaps the most compelling specification is the 240W power envelope for the full PCIe AIC card. To put this in perspective, a single NVIDIA H100 GPU draws approximately 700W under load, while the newer B200 can consume up to 1,000W. Running a 700B-parameter model on GPU-only infrastructure typically requires multiple high-end GPUs, pushing total system power consumption into the multi-kilowatt range.

Skymizer's approach of offloading the decode phase to a dedicated 240W accelerator could yield substantial energy savings at scale. Consider the math for a typical data center deployment:

• GPU-only setup for 700B model: 8x H100 GPUs at ~700W each = ~5,600W
• Hybrid setup: Fewer GPUs for prefill + HTX301 AIC for decode = potentially significant power reduction
• Cooling savings: Lower power draw means less cooling infrastructure
• TCO impact: Reduced electricity and cooling costs compound over 3-5 year deployment cycles

While exact performance benchmarks and real-world comparisons have not yet been published, the power efficiency claims alone make the HTX301 worth watching. Energy costs now represent one of the largest operational expenses for AI infrastructure operators, and any meaningful reduction translates directly to the bottom line.

The Disaggregated Inference Trend Gains Momentum

Skymizer is not alone in recognizing the opportunity to disaggregate AI inference into specialized phases. The broader industry has been moving toward this model, with several startups and established players exploring similar approaches.

Groq, for instance, has built its LPU (Language Processing Unit) architecture specifically for inference workloads, achieving impressive tokens-per-second performance. Cerebras takes a different approach with its wafer-scale engine but similarly targets inference efficiency. And companies like SambaNova and Graphcore (now part of SoftBank) have also pursued non-GPU architectures for AI workloads.

What distinguishes Skymizer's approach is the explicit focus on the decode phase alone, rather than attempting to replace GPUs entirely. This 'complementary' rather than 'competitive' positioning could prove strategically smart — it lowers the barrier to adoption since customers do not need to abandon their existing GPU investments.

The prefill-decode disaggregation model also aligns with emerging software frameworks. Projects like vLLM, TensorRT-LLM, and various serving frameworks already support splitting inference workloads across different hardware. This software ecosystem readiness could accelerate adoption of specialized decode accelerators like the HTX301.

What This Means for Developers and Businesses

For organizations deploying large language models, the HTX301 represents a potentially significant shift in how inference infrastructure is architected. Here are the practical implications:

• Lower barrier to large model deployment: 384GB of memory on a single card means 700B-parameter models become accessible without massive multi-GPU clusters
• Improved GPU utilization: By offloading decode work, existing GPUs can focus on prefill and handle more concurrent requests
• Energy cost reduction: The 240W power envelope could dramatically reduce per-inference energy costs
• Edge and on-premises viability: The compact PCIe form factor makes local deployment of very large models feasible for enterprises with data sovereignty requirements
• Hybrid architecture flexibility: Organizations can mix and match GPUs for prefill with HTX301 cards for decode, optimizing cost-performance ratios

Developers working with inference serving frameworks should watch for SDK and integration announcements from Skymizer. The value of the hardware will ultimately depend on software ecosystem support, including compatibility with popular frameworks like PyTorch, vLLM, and Triton Inference Server.

Looking Ahead: Challenges and Open Questions

While the HTX301 specifications are impressive on paper, several questions remain unanswered. Skymizer has not yet disclosed detailed performance benchmarks — specifically, tokens-per-second throughput during the decode phase, which is the metric that matters most for real-world deployment.

Pricing is another unknown. The economics of adding a dedicated decode accelerator only make sense if the total cost of ownership is lower than simply adding more GPUs. Without pricing information, it is difficult to assess the true value proposition.

There is also the question of software maturity. NVIDIA's dominance in AI hardware is as much about its CUDA ecosystem and software stack as it is about raw silicon performance. Skymizer will need to demonstrate that its LISA architecture and HyperThought platform can deliver a developer experience that is smooth enough to justify the learning curve.

Finally, the competitive landscape is intensifying rapidly. Custom AI silicon from hyperscalers — including Google's TPUs, Amazon's Trainium and Inferentia chips, and Microsoft's Maia — represents formidable competition. These companies have both the scale and the captive workloads to drive adoption of their own inference hardware.

Despite these challenges, Skymizer's HTX301 represents an intriguing bet on the future of disaggregated AI inference. As models continue to grow in size and inference demand scales exponentially, the case for specialized, energy-efficient decode hardware will only strengthen. The company's next steps — publishing benchmarks, announcing partnerships, and revealing pricing — will determine whether the HTX301 becomes a meaningful player in the AI infrastructure stack or remains a niche curiosity.