log 019

Dev Log: Timelines

I'm building a DAW in the browser using the web audio API. If you have no idea what that means no worries, here is a picture that will help.

There is a LOT that goes into building a DAW, but one of the foundations is the Timeline. The area where you actually put audio clips and arrange the song along. Timelines are not unique to editing music though! They are also a fundamental building block in video editing, animations, and all sorts of interesting projects.

BUT there is no chance I can cover everything in this log. Let's walk through the anatomy of a timeline first, and then we can breakdown how to build the three pieces that we need for any timeline based project we might build.

  1. Navigator
  2. Zoom Window
  3. Projection

The Navigator represents the full length of your content. Whether that is a song, a video, or a random grid of squares. The whole use case for a timeline is when you are working with data that cannot be displayed all at once and you want to provide a visual form of navigating the content and controls for changing how dense the content you are currently displaying is.

The Zoom Window is the control mechanism for doing the actual work of navigating and changing displayed content density.

The Projection displays an editable window of the content that is currently the target of the zoom window. As the zoom window changes location or size so does the projection of that data into this window.

Let's start by getting all of the UI elements we need on the screen. Then we will go step by step to get them working.

export function Timeline() {
  return (
    

      
      
    

  )
}

function Navigator() { 
  return ( 
    

      

        {/* Zoom Window */}
        

          {/* Left handle */}
          

          {/* Right handle */}
          

        

      

    

  ) 
}

function Projection() { 
  return (
    

      

        

        {/* Content goes here */}
      

    

  )
}

Normally I would just put everything into a single component at this point, buuut I know how much code we are about to be adding so I saved us some trouble and went ahead and split things up.

This line is important:

overscrollBehaviorX: 'none'

This avoids the default behavior in macOS where scrolling left on trackpad moves you to the previous url.

Now let's add the state needed to make our static timeline something we can control.

Our core timeline state consists of 3 values.

  • Content length: this is the full length of the content in the timeline
  • View: this is an object that represents the amount of the content we are currently editing
  • Min: This is the smallest length either our content or view can be

For all of our data we will be building out modules that define the types and operations available for it. This keeps our core logic framework agnostic.

Let's start with our Timeline type.

// timeline.ts
import type { Numeric } from './numeric'
import * as S from './span'

export type Timeline = {
  size: A
  view: S.Span
  min: A
}

Then let's add a function that we will use when we first create our Timeline state. This function is important because it's how we make guarantees that our view is always within the bounds of the min length and the full length of our content.

export function normalize(
  N: Numeric,
  t: Timeline,
): Timeline {
  const size = N.max(t.size, N.zero)
  const min = N.clamp(t.min, N.zero, size)

  const viewSize = N.clamp(t.view.size, min, size)
  const maxStart = N.subtract(size, viewSize)
  const start = N.clamp(t.view.start, N.zero, maxStart)

  return { size, min, view: { start, size: viewSize } }
}

Now we can use it in our timeline component.

export function Timeline() {
  const [timeline, setTimeline] = useState>(() => ({
    size: Px.Px(20000),
    min: Px.Px(200),
    view: S.make(Px.Numeric, 2000, 4000),
  }))

  return (
    

      
      
    

  )
}

We have now ran into our first gotcha when working with Timelines. How do you render a content that is 20000 pixels? All the numbers we are storing right now are content pixels. They represent the literal size of the data. That means if we tried to take them and use them directly to render to the dom everything would be WAY too big.

All of this data needs to be scaled based on the available room we have in our DOM elements. How do we do that?

Well if we only have 1800px available in the DOM that means our 20000 = 1800. What does that mean for our min value? And what about the positions along it where our zoom window is located? Well, let's take a little stroll back to middle school algebra class.

In school this was called cross multiplying or proportional fractions. However, we are going to call this projections in our context and build a module around it. Its focus is on converting content pixels to screen pixels and vice versa.

// projection.ts
import type { Numeric } from './numeric'
import * as Px from './px'

export function scaleFor(
  N: Numeric,
  size: A,
  width: Px.Px,
): number {
  return N.make(width / size)
}

export function toScreen(
  N: Numeric,
  from: A,
  at: A,
  scale: number,
): Px.Px {
  return Px.Px(N.subtract(at, from) * scale)
}

export function fromScreen(
  N: Numeric,
  from: A,
  at: Px.Px,
  scale: number,
): A {
  return N.make(from + at / scale)
}

Alrighty, we have the conversion functions, but if you take a look at the implementation there are two new pieces of data we don't have in our implementation yet.

The scaleFor function is how we get the first piece of data we need so that we can call the to/from functions, but to get the scale we need the container width. For that we need to go to the DOM and grab it from there directly once it renders.

To do that we can use this handy hook. It allows you to track an element's size and make sure it stays updated even if the element resizes due to outside factors like browser width change.

export function useElementSize() {
  const ref = useRef(null)
  const [size, setSize] = useState({
    width: 0,
    height: 0,
    x: 0,
    y: 0,
    bottom: 0,
    left: 0,
    right: 0,
    top: 0,
  })

  useIsomorphicLayoutEffect(() => {
    const el = ref.current
    if (!el) return

    const ro = new ResizeObserver((entries) => {
      const cr = entries[0]?.contentRect
      if (!cr) return
      setSize({
        width: cr.width,
        height: cr.height,
        x: cr.x,
        y: cr.y,
        left: cr.left,
        right: cr.right,
        top: cr.top,
        bottom: cr.bottom,
      })
    })

    ro.observe(el)
    return () => ro.disconnect()
  }, [ref])

  return { ref, size }
}

Then we use this hook to calculate our projections so that our zoom window can be based off our actual timeline state instead of static values.

function Navigator({ timeline, setTimeline }: Props) { 
  const { view } = timeline
  const { ref, size } = useElementSize()

  const scale = useMemo(() => {
    return Projection.scaleFor(Px.Numeric, timeline.size, Px.Px(size.width))
  }, [size.width, timeline.size])

  const zoomWindow = useMemo(
    () => Span.transform(view, (v) => 
      Projection.toScreen(Px.Numeric, Px.Numeric.zero, v, scale)
    ),
    [scale, view],
  )

  return (
    

      

        {/* handles... */}
      

    

  )
}

Now our zoom window is based on our actual state values. If those change then our zoom window will change position and size too….but we don't have any way to make those changes yet.

We need to add some actions that you would expect out of a timeline navigator:

Let's take these one by one starting with the event that is attached to the element highest in the tree (snap) and add the DOM events and listeners we need.

One common thread between all of these actions is being able to get the position of the cursor within the navigator. Since the timeline runs only along one dimension we are grabbing the cursor position as an offset from the left side of the navigator along the X axis.

import { clamp } from '../lib/math'

export function getPointerPosition(
  e: React.PointerEvent,
  element?: HTMLElement | null,
): { x: number; y: number } {
  if (!element) return { x: e.clientX, y: e.clientY }
  const rect = element.getBoundingClientRect()
  const x = clamp(e.clientX - rect.left, 0, rect.width)
  const y = clamp(e.clientY - rect.top, 0, rect.height)
  return { x, y }
}

With that helper we can kick things off by adding the ability to pan a timeline in the core modules. This requires a change in our core files. Panning is moving our view around within the bounds of the timeline. Since our view is a Span type, we can extract the Span type into its own module and handle the move logic there.

// span.ts
import type { Numeric } from './numeric'

export type Span = {
  start: A
  size: A
}

export function move(
  N: Numeric,
  s: Span,
  delta: A,
): Span {
  return { start: N.add(s.start, delta), size: s.size }
}

export function center(N: Numeric, s: Span): A {
  return N.add(s.start, N.divide(s.size, 2))
}

export function end(N: Numeric, s: Span): A {
  return N.add(s.start, s.size)
}

export function transform(
  s: Span,
  f: (a: A) => A,
): Span {
  return { start: f(s.start), size: f(s.size) }
}

Then inside of our Timeline module we will move our view and run it through our normalize function to enforce that the changes we made fall within the bounds of the timeline.

export function panBy(
  N: Numeric,
  t: Timeline,
  delta: A,
): Timeline {
  const tt = normalize(N, t)
  const view = S.move(N, tt.view, delta)
  return normalize(N, { ...tt, view })
}

Now that our core functionality exists, let's add our event handler. We are going to do this in two layers. Layer one is the react layer. It handles checking for refs, and any event specific handling that needs to be done.

The other layer is for calculating the delta required from the pan function in the timeline module and returning the adjusted timeline. We want to make the offset a parameter. For snapping we want to use a hard coded value representing the center point of the zoom window, but for other interactions like panning we will have a dynamic pointer offset across the zoom window.

function deltaFrom(
  N: Numeric,
  { x, scale, offset, from }: {
    x: Px.Px
    offset: A
    scale: number
    from?: A
  },
): A {
  // Convert screen position to timeline position
  const at = Projection.fromScreen(N, N.zero, x, scale)
  // nextStart = pointer timeline position - offset from zoom window edge
  const nextStart = N.subtract(at, offset)
  // delta = how much to move from current view.start
  return N.subtract(nextStart, from ?? N.zero)
}

Once you have snap setup the next interaction to tackle is panning the zoom window around the timeline.

However, there are two big differences in how we handle panning.

The first difference means we need to make sure events do not propagate up the tree using stopPropagation

We need pointer capturing, and we need data that gets passed around the full event lifecycle.

If you are not familiar with setPointerCapture we will be using it quite a bit. It's an extremely useful browser feature especially for drag based interfaces. What we are telling the browser by using it is: "No matter where my mouse goes on the page. As long as I still have the pointer captured send events to this element." This means if our cursor drags outside of the navigator our events don't suddenly cutoff and stop working.

As for what data is needed across the lifecycle we will need to track the following:

Our ref data will go in a new model we can call interaction, and we can set up a type for the data we need while panning.

type Direction = 'L' | 'R'
type Idle = { kind: 'idle' }
type Pan = {
  kind: 'pan'
  initialTimeline: Timeline.Timeline
  offset: A
}
type Resize = {
  kind: 'resize'
  initialTimeline: Timeline.Timeline
  direction: Direction
}

type Interaction = Idle | Pan | Resize

Here is the full implementation of the pan events:

const interactionRef = useRef>({ kind: 'idle' })

const handlePanStart = (e: React.PointerEvent) => {
  if (!ref.current) return
  e.preventDefault()
  e.stopPropagation()

  ref.current.setPointerCapture(e.pointerId)
  const { x } = getPointerPosition(e, ref.current)
  const offset = deltaFrom(Px.Numeric, {
    x: Px.Px(x),
    scale,
    offset: Px.divide(timeline.view.size, 2),
  })
  interactionRef.current = { kind: 'pan', initialTimeline: timeline, offset }
}

const handlePan = (e: React.PointerEvent) => {
  if (
    interactionRef.current.kind !== 'pan' ||
    !ref.current ||
    !ref.current.hasPointerCapture(e.pointerId)
  ) {
    return
  }

  const pointer = getPointerPosition(e, ref.current)
  const delta = deltaFrom(Px.Numeric, {
    scale,
    x: Px.Px(pointer.x),
    offset: interactionRef.current.offset,
    from: timeline.view.start,
  })
  const nextTimeline = Timeline.panBy(Px.Numeric, timeline, delta)
  setTimeline(nextTimeline)
}

const handleInteractionEnd = (e: React.PointerEvent) => {
  if (!ref.current || !ref.current.hasPointerCapture(e.pointerId)) {
    return
  }
  ref.current.releasePointerCapture(e.pointerId)
  interactionRef.current = { kind: 'idle' }
}

Time for our next set of events. These are a pair of events for resizing the zoom window by dragging on an edge of the window.

const handleResizeStart = (direction: Direction) => (e: React.PointerEvent) => {
  if (!ref.current) return
  e.preventDefault()
  e.stopPropagation()

  ref.current.setPointerCapture(e.pointerId)
  interactionRef.current = {
    kind: 'resize',
    direction,
    initialTimeline: timeline,
  }
}

const handleResize = (e: React.PointerEvent) => {
  if (
    interactionRef.current.kind !== 'resize' ||
    !ref.current ||
    !ref.current.hasPointerCapture(e.pointerId)
  ) {
    return
  }

  const { x } = getPointerPosition(e, ref.current)
  const pointerTimelinePos = Projection.fromScreen(
    Px.Numeric,
    Px.Numeric.zero,
    Px.Px(x),
    scale,
  )

  if (interactionRef.current.direction === 'L') {
    const delta = Px.subtract(pointerTimelinePos, timeline.view.start)
    const nextTimeline = Timeline.resizeLeftBy(Px.Numeric, timeline, delta)
    setTimeline(nextTimeline)
  } else {
    const delta = Px.subtract(
      pointerTimelinePos,
      Span.end(Px.Numeric, timeline.view),
    )
    const nextTimeline = Timeline.resizeRightBy(Px.Numeric, timeline, delta)
    setTimeline(nextTimeline)
  }
}

The final piece of the puzzle is the ability to change the zoom window size from the center point. The way we are going to approach this is sharing the pointer event with the panning functionality, but with two adjustments.

The reason we want to require alt key is that no human can perfectly drag a mouse along a single axis. There are many options for how to handle differentiating between pan and zoom, but I like the clarity of having the alt key activate the alternate functionality.

// In timeline.ts
export function zoomAt(
  N: Numeric,
  t: Timeline,
  factor: number,
  anchor: A,
): Timeline {
  const tt = normalize(N, t)
  const a = N.clamp(anchor, N.zero, tt.size)

  if (N.eq(tt.view.size, N.zero)) {
    const nextSize = N.divide(tt.view.size, factor)
    return normalize(N, { ...tt, view: S.withSize(tt.view, nextSize) })
  }

  const nextSizeRaw = N.divide(tt.view.size, factor)
  const nextSize = N.clamp(nextSizeRaw, tt.min, tt.size)

  // anchorT in [0..1] relative to current view
  const anchorT = N.divide(N.subtract(a, tt.view.start), tt.view.size)

  // nextStart = anchor - anchorT * nextSize
  const nextStart = N.subtract(a, N.multiply(anchorT, nextSize))

  return normalize(N, { ...tt, view: S.make(N, nextStart, nextSize) })
}

// Usage with scrub hook
const { scrubProps: zoomScrubProps, isScrubbing } = useScrub({
  onScrubStart: () => {
    initialTimelineRef.current = timeline
  },
  onScrub: (delta) => {
    const factor = factorFromDelta(delta, rate)
    const nextTimeline = Timeline.zoomAt(
      Px.Numeric,
      initialTimelineRef.current,
      factor,
      Span.center(Px.Numeric, initialTimelineRef.current.view),
    )
    setTimeline(nextTimeline)
  },
})

function factorFromDelta(dy: number, rate = 350): number {
  return Math.pow(2, dy / rate)
}

To track when the alt key is being held we can use a simple hook to update pointer style:

export function useGlobalKeyPressed(key: string): boolean {
  const [isPressed, setIsPressed] = useState(false)
  
  useEffect(() => {
    const handleKeyUp = (e: KeyboardEvent) => {
      if (e.key === key) setIsPressed(false)
    }
    const handleKeyDown = (e: KeyboardEvent) => {
      if (e.key === key) setIsPressed(true)
    }

    window.addEventListener('keyup', handleKeyUp)
    window.addEventListener('keydown', handleKeyDown)
    return () => {
      window.removeEventListener('keyup', handleKeyUp)
      window.removeEventListener('keydown', handleKeyDown)
    }
  }, [key])
  
  return isPressed
}

// Usage
const isAltKeyPressed = useGlobalKeyPressed('Alt')



  {/* zoom window content */}

Projection Sync

The last piece of the puzzle is syncing the Projection component with the timeline state. The Projection is a scrollable container that shows the content within the current view. When the navigator changes the view, the projection should scroll to match, and when the user scrolls the projection, the timeline state should update.

function Projection({ timeline, setTimeline, isInteracting = false }: Props) {
  const suppressScrollEventsRef = useRef(false)
  const { ref, size } = useElementSize()

  const scale = useMemo(() => {
    return P.scaleFor(Px.Numeric, timeline.view.size, Px.Px(size.width))
  }, [size.width, timeline.view.size])

  const contentWidth = useMemo(() => {
    return Scroll.width(Px.Numeric, timeline.size, scale)
  }, [timeline.size, scale])

  // state -> DOM
  useEffect(() => {
    if (!ref.current || isInteracting) return
    const nextScrollLeft = Scroll.toScroll(Px.Numeric, timeline.view.start, scale)
    if (Math.abs(ref.current.scrollLeft - nextScrollLeft) < 0.5) return

    suppressScrollEventsRef.current = true
    ref.current.scrollLeft = nextScrollLeft
    requestAnimationFrame(() => {
      suppressScrollEventsRef.current = false
    })
  }, [scale, timeline.view.start, isInteracting])

  // DOM -> state
  const onScroll = (_: React.UIEvent) => {
    if (suppressScrollEventsRef.current || !ref.current) return

    const nextStart = Scroll.fromScroll(
      Px.Numeric,
      Px.Px(ref.current.scrollLeft),
      scale,
    )

    const nextTimeline = Timeline.panBy(
      Px.Numeric,
      timeline,
      Px.subtract(nextStart, timeline.view.start),
    )

    setTimeline(nextTimeline)
  }

  return (
    

      

      {/* Content rendered here */}
    

  )
}

Notice the bidirectional sync here. We have state → DOM (updating scroll position when timeline changes from navigator) and DOM → state (updating timeline when user scrolls the projection). The suppressScrollEventsRef prevents infinite loops by ignoring scroll events triggered by our own state updates.

And there we have it! A fully functional timeline system with navigator, zoom window, and projection. You can check out the live demo to see it in action.

I hope you enjoyed

There is a lot more coming...

If you want to get updates when I publish new guides, demos, and more just put your email in below.